Non-autoregressive Text Editing with Copy-aware Latent Alignments
Authors: Yu Zhang
Yue Zhang
Leyang Cui
Guohong Fu
Venue: EMNLP 2023 Main (2023)
Keywords: text generation
text editing
grammatical error correction
sentence fusion

Abstract: Recent work has witnessed a paradigm shift from Seq2Seq to Seq2Edit in the field of text editing, with the aim of addressing the slow autoregressive inference problem posed by the former. Despite promising results, Seq2Edit approaches still face several challenges such as inflexibility in generation and difficulty in generalizing to other languages. In this work, we propose a novel non-autoregressive text editing method to circumvent the above issues, by modeling the edit process with latent CTC alignments. We make a crucial extension to CTC by introducing the copy operation into the edit space, thus enabling more efficient management of textual overlap in editing. We conduct extensive experiments on GEC and sentence fusion tasks, showing that our proposed method significantly outperforms existing Seq2Edit models and achieves similar or even better results than Seq2Seq with over $4\times$ speedup. Moreover, it demonstrates good generalizability on German and Russian. In-depth analyses reveal the strengths of our method in terms of the robustness under various scenarios and generating fluent and flexible outputs.

Non-autoregressive Text Editing with Copy-aware Latent Alignments
Yu Zhang Yue Zhang Leyang Cui
Guohong Fu
Institute of Artificial Intelligence, School of Computer Science and Technology,
Soochow University, Suzhou, China
Tencent Al Lab
y www..sssooooooooo leyangcui@tencent. com; ghfu@suda edu.
loetangg.ssltllooo.cooom com; yzhang218stu. aanda.edus
httpp://ggthub...c//yhaaggge www.////aaagsssctttoooeoooo
Abstract
Grammatical Error Correction
source:
Me want go store.
Recent work has witnessed paradigm shift from Seq2Seq to Seq2Edit in the field of text editing, with the aim of addressing the slow autoregressive inference problem posed by the former. Despite promising results, Seq2Fidit approaches still face several challenges such as inflexibilityy in generation and difficulty in gen- eralizing to other languages. In this work, we propose a novel non-autoregressivee text edit- ing method to circumvent the above issues, by modeling the edit process with latent CTC alignments. make a crucial extension to CTC by introducing the copy operation into the edit space, thus enabling more efficient management of textual overlap in editing. We GEC and
target: I want to go the store.
Sentence Fusion
source:
The sun The sky darkened.
target:
As the sun set, the sky darkened.
Table I: Text editing examples for grammatical error correc- tion and sentence fusion.
autoregressive generation fashion, i.e., generating tokens one by one. Consequently, the practical ap- plications of Seq2Seq models are limited in modern online text assistance systems.
To overcome the above deficiency, recently, there is a growing interest in an alternative ap- proach, referred to Seq2Edit (Awasthi et al., 2019; Omelianchuk al., 2020; Mallinson et al., 2020), which, in contrast, proposes to reconstruct the target sentence by applying a set of edit op- erations, e-g., keep, deletion and insertion, the input. Drawing on the insight that the input/output tokens are heavily shared, Seq2Edit favors copy- ing most of the source text directly via the keep operation, which eases the reliance for autore- gressive decoder (Malmi et al., 2019; Mallinson et al., 2022). Among others, the best- test-performing GECToR (Omelianchuk et al., 2020, 2021) directly formulates text-editing as a non non-autoregressive se- quence tagging task, thus enabling more efficient parallelizable inference. GECToR demonstrates re- markable results on many tasks, meanwhile being orders of magnitude faster than its autoregressive counterparts (Rothe et al., 2020).
conduct extensive experiments sentence fusion tasks, showing that our pro- posed method significantly outperforms exist- ing Seq2Edit models and achieves similar or even better results than Seq2Seq with over 4x speedup. Moreover, it demonstrates good gen- cralizability on German and Russian. In-depth analyses reveal the strengths of our method in terms of the robustness under various scenarios and generating fluent and flexible outputs.
1 Introduction
In natural language processing, monolingual text generation involves producing target sequence from the source text with significant textual overlap (Malmi et al., 2022). This includes range of text- editing tasks such as grammatical error correction (GEC) (Ng al., 2014) and sentence fusion (Geva et al., 2019), as shown in Table
Generally, text editing can be addressed under the standard Seg2Seq framework (Lehanoff et al., 2020; Rothe et al., 2021). Despite their decent per- formance, Seq2Seq has been criticized (Sun et al., 2021) for its inferior inference speed due to the
However, several challenges arise as we try have the cake and cat it. We argue that Scq2Edit works represented by GECToR still suffer from two main issues:
Flexibility: Seq2lidit learns to edit text by pre- defining a fixed and relatively small (e.g., 5,000) edit vocabulary collected from the training data, which is at the sacrifice of generation flexibility.
"Work was done during the intersship at Tencent Al Lab. Zhang and Yue Zhang make equal contributions. Corresponding author.

---

ii Language generalization: Seq2Eddit needs to delve into linguistic features to customize the edit actions, e.g., VB-VEZ for subject-agreement edits and PLURAL for singular plural form con- versions, thus diminishing its ability to general- ize to other languages.
We show that our model achieves similar or even better results compared Seq2Seq across all experiments with ~1x speedup.
Extensive analyses on our method reveal its mer- its in terms of robustness under different sce- narios as well as the superiority in generation fiexibility against existing systems.
Our desiderata in this work is to design non- autoregressive model for text editing that enjoys the merits of both efficiency and effectiveness, meanwhile generalizing well to other languages. This poses two considerations: 1) flexible, non- manually defined edit space; 2) a minimal set of tailored operations (Dong et al., 2019) to main- tain the generalization. Taking inspirations from recent progresses in non-autoorerressvve text gen- eration (Libovický and Helcl, 2018; Saharia et al., 2020; Huang ct al., 2022b), in this work, we pro- pose novel method for text editing that meets the aforementioned expectations by making a direct yet effective extension to connectionist temporal classification (CTC) (Graves et al., 2006).
2 Preliminaries
We begin by introducing some notations. The goal for text-editing is to transform the source sentence x 20, .1,... EN into the desired target V1, UM with N and M tokens, respectively.
Connectionist Temporal Classification first introduced in auto speech recognition (ASR) (Graves al., 2006), aiming to circumvent the problems no explicit alignments between ASR inputs/outputs. Specifically, CTC introduces a spe- cial blank token Ø on top of the vocabulary V, and defines a latent aligmment path a = a1..... AN between æ and y with a; VU(Ø), which is of equal length
Unlike previous works focusing on generating ar- bitrary tokens (Gu and Kong, 2021), the key insight here interpret the vanilla CTC alignment as an executable edit sequence, primarily composed of two kinds of operations: DELETE and ADD, This perspective opens the door for combining the align- ment with the edit actions in existing Scq2Edit works. Specifically, we further extend the align- ment space by incorporating KEEP, a label used to facilitate direct copy of the respective source to- kens. We find it is essential for processing textual overlap in editing, yielding significant performance gains. During training, our method marginalizes out all (valid) latent edit alignments to maximize the likelihood of the target text (Graves et al., 2006). During inference, like GECTOR, it simply takes the token with highest probability the output for each position simultancously (see Table 2), ensur- ing the high efficiency. The contributions of this work are four-fold:
During training, CTC models the probability of the target sequence by marginalizing the probabilities of all latent alignments,
P( x) = Eacrio) P(a x)
(1)
where I(-) is mapping function that returns all possible aligmment paths. CTC views each a, E independent of each other and factorizes the proba- bility of a as
P(a x) = II La:ca P(a. x))
(2)
In this way, CTC permits very efficient calcula- tion of Eq. in O(N x M) via forward algorithm. We refer interested readers to the original paper (Graves et al., 2006) and tutorials by Hannun (2017) for more details.
During inference, CTC defines collapsing func- tion (-)(-) to recover the target sequence y from by removing all blanks and consecutive repet- itive tokens. For example, assuming a possible alignment path a = {a, Ø, b, b), then r (aa) returns {a, a, b).
We propose a novel method, that extends CTC with the copy operation to address edit-based text generation. To the best of our knowledge, this is the first attempt to adapt CTC to deal with text editing tasks.
Non-autoregressive text generation (NAT) dif- fers from its autoregressive counterpart in that it generates all target tokens simultancously rather than one-by-one. NAT often runs several times faster than autoregressive Seq2Seq models as it is highly parallelized. Very recently, CTC has been
We conduct experiments on GEC and sentence fusion, and find that our proposed model per- forms favorably better than all existing Seq2Edit models, meanwhile showcasing good generaliza- tion capabilities in multilingual settings.

---

dog
like
an
dog
K
K
K
Ø
2
2
dogs
2
like
dogs
V
I
like
dogs
Table 2: inference example for editing the source sentence a. The output is produced by our copy-aware CTC, which predicts an I-best token for each position with the green label denoting the copy label. The output y is the final recovered result by the collapsing function (-).
Ø
like
where each r; E IH, H is the size of the hidden vector. Once the hidden states are obtained, we employ a simple linear projection followed by two Transformer decoder layers to upsample each h; to T new sample vectors, ensuring that the scaled input, which is T' as long as the the source, is strictly longer than the desired output
dogs
Ø
Figure running GEC example by CTC with an upsam- pling ratio of 2. which corrects phase "an dog" in the source sentence to "dogs". Each source token is separated by red-dashed lines. Grey black nodes represent blanks (2) and normal tokens respectively. Green nodes indicate the positions where source tokens can be copied directly. The arrows represent all valid transition paths.
h 1......... h T Decoder (Wr; (4)
where W RT TH learnable pa- rameters. We fix the value of upsampling ratio T to 4 in this work after careful ablation analyses ($ 5.1). In this way, we can generate target sen- tences by CTC with very flexible length control. We employ another linear layer followed by the softmax function over h; to obtain P(a: Φ).
introduced non-autoregressive NMT (Libovický and Helcl, 2018; Saharia et al., 2020; Gu and Kong, 2021). In ASR, CTC assumes the input length N is larger that the output length M so that we can safely delete blanks from the alignment, resulting in shorter output sequence. However, this is not the fact in text generation. To remedy this, Li- of
bovický and Helel (2018) propose to make upsampling layer amplify the input first and then run CTC as usual. This enables the model to learn target lengths very flexibly, which we be- lieve is an important factor that empowers the CTC model.
3.2 Copy-aware CTC
The output space of vanilla CTC comprises the general vocabulary as well as the blank token 2. We can utilize CTC to mimic the edit processes by symbolizing generating token E V as ADD, rep- resenting the insertion operation, and Ø as DELETE, meaning deleting a source token. This satisfies the aforementioned desiderata of learning to edit with minimal set of operations (Dong et al., 2019). and maintaining enough flexibility by means of marginalizing all latent alignments defined over the entire vocabulary. However, vanilla CTC still wasteful for text editing as it lacks explicit model- ing of the copy behavior. We in this work propose to bridge this gap by introducing special token to denote the KEEP operation. Concretely, we interpret generating at arrtj. the jth upsampled position for ith source token, as directly copying the source token Ty. this way, the final output space of each a; is V U(k] U(ø).
3 Methodology
In this section, we will introduce our proposed method that adapts the vanilla CTC to text editing tasks. The main idea is to endow CTC with the ability of modeling the edit processes by extending the latent CTC alignments with interpretable edit operations, especially the copy operation.
3.1 Model
The basic architecture of our model is encoder-only. Given the input take pretrained language model (PLM) (Devlin al., 2019) as the backbone encoder to obtain the contextualized representations.
22, ... NN, simply
Training objective Our final objective is to min- imize the negative log-likelihood of all possible
T1, T2...... TN = PLM 11, T....., ..),
(3)

---

alignments with the three kinds of edit operations
brought considerable performance gains, but the improvements saturate gradually after iterations. So we choose to uniformly refine the outputs twice for a good pereerdaefoomnncce tradeoff.
C -og log
Σ (v)
P(a
x))
(5)
where (-)) is the new mapping function extending T(.) to KEEP. Assuming an input {a, a, b) with T 2, two possible paths returned by (-) can be a, 00, a, b, b) and {K, K, K,K.
Experiments
Following FELIX and EDIT5 (Mallinson et al., 2020, 2022), we evaluate our model by conducting experiments two text editing tasks: grammatical error correction (GEC) and sentence fusion, both of which are representative and have sufficient data for training. We plan to conduct examinations on more tasks in future work due to space limitations.
Glancing training Previous works have shown that the glancing training strategy (Qian et al., 2021) can give boost the performance of non- autoregressive generation. So we also adopt this method in our training process. The key idea is to sample some ground-truth tokens as inputs to the decoder to guide the model once the references are too difficult to fit, which is reminiscent of curricu- lum learning. Specifically, the objective becomes
4.1 Grammatical Error Correction
The task of grammatical error correction involves detecting and correcting the grammatical errors in a given sentence.
Cglancing log P(y æ h)
Setup For English, we adopt a 3-stage training strategy to train our GEC models (Zhang et al.. 2022a): 1) pretrain the model on CLANG-8 (Rothe et al., 2021), a cleaned version of the LANG-8 data; 2) finetune the pretrained model on the combina- tion of three datasets, namely FCE (Yannakoudakis et al., 2011), NUCLE (Dahlmeier et al., 2013) and W&I+LOCNESS (Bryant al., 2019); 3) finally, further finetune the model on the high-quality W&I+LOCNESS. During training, we use BEA19 Dev data as the validation set. We evaluate our models by reporting P/R/Fo.s points on BEA 19 Dev data using the ERRANT toolkit (Bryant et al.. 2017) and CoNLL14 Test data (Ng et al., 2014) using M2Scorer (Dahlmeier and Ng, 2012). Be- sides, without additional training, GLEU scores JFLEG Test data (Napoles et al., 2017) to measure the fluency of CTC-generated texts. More details on data statistics and training details are available in $ A.
where h is obtained by replacing some hidden states h. with the embeddings of sampled gold tokens. The sampling ratio determined following three steps (Qian et al., 2021): 1):1111 1) determine a gold alignment compatible with the target by Viterbi decoding (Chan et al., 2020; Huang et al., 2022b; Shao et al., 2022): a arg maX P(a | w); 2) decode an 1-best alignment path a' 3) then the ratio becomes p [a a'] which is pro- portional to the number of different tokens between the predicted sequence and the gold alignment. We set r to in the following experiments.
3.3 Inference
During inference, we directly predict the 1-best token for each position in parallel. followed by the post-processing process. Taking Table an example, first, an alignment path a' = {K, K, K, Ø, Ø, Ø, dogs) is produced by finding the 1-best token for each position greedily. Then each label is translated to the corresponding source token. Finally, we can successfully re- cover the output with the collapsing function, i.c., I' (aa))= = r --, I, like, like, Ø, Ø, Ø, dogs)) = (I, like, dogs3.
also report
Results We present the main GEC results in Ta- ble 3. is hard to make fully fair comparisons with existing works as the training data varies vastly, which has a huge impact on the final re- sults (Omelianchuk et al., 2020). We therefore re- re-implemented BART-based Seq2Seq and GEC- ToR and ran them under the same environments for more comparable results. In the top group of the Table, first, we observe that our re-implemented BART achieves 68.2 Fo.5 score on CoNLLL14 out- performing the cutting-edge SynGEC (Zhang et al., 2022b); second, our CTC is superior to BART
Iterative decoding Following Omelianchuk et al. (2020), also employ the techniques of iterative decoding better capture the edits hard to make in one pass. We simply take the collapsed output CTC as the model input during the next iteration (Awasthi et al., 2019). In pratice, we found that

---

BEA19
CoNLL14
PL.Ms
JFLEG
Specdup
R
F
P
R
F
Autovegressive
Kaneko et al.
69.3
45.0
62.6
61.3
SAD (Sun et al.)*
BART
71.0
52.8
66.4
3.8x
Seq2Seq (Zhang et al.)
BART
63.1
44.8
58.3
73.6
48.6
66.7
61.5
SynGEC (Zhang al.)
BART
64.5
45.7
59.6
74.7
49.0
67.6
62.2
ChatGPT (Fang et al.)
51.3
62.4
63.2
63.5
Seq2Seq'
BART
65.0
40.7
58.0
76.0
48.4
68.2
60.2
1.0%
nonnaudooeegeessvve
LaserTagger (Malmi et al.)
BERT
50.9
26,9
43.2
GECToR (Omelianchuk et ) GECToR Ours
RoBERTa
44.4
66.1
RoBERTa
66.2
36.5
56.9
73.2
51.1
67.4
57.6
2.9%
RoBERTa 62.2
46.9
58.4
74.9
50.6
68.3
62.4
4.1x
Table 3: Main results on BEA19 Dev, CoNLL14 Test, and JFLEG Test data. random seeds. means using external or synthetic data for pretraining: uses external syntactic knowledge, and thus incomparable. means the results are ohtained from running re-implemented code.
Our results are averaged over runs with different
all datasets by 0.4, 0.1 and 2.2, respectively. On BEA19 Dev data, CTC surpasses all previous works except SynGEC, which enhanced by exter- nal syntax trees. On JFLEG Test data, our model exhibits a GLEU score 62.4, second only to Chat- GPT (Fang et al., 2023), which is very close to human-like performance, indicating that our model excels at generating fluent sentences. In the bottom group, see that our CTC greatly surpasses the best-performing GECToR by 1.5, 0.9 and 4.8 on BEA19 Dev, CoNLL14 Test and JFLEG Test data, respectively, achieving new state-of-the art in the area of non-autoregressive text-editing.
EM
SARI
Autoregressive
Transformer (Geva et al.)
51.1
84.5
LaserTaggerae (Malmi al.)
53.8
85.5
Seq2Edits (Stahlberg and Kumar)
61.71
88.73
BERT stme (Rothe al.)
65.3
89.9
RoBERTaaue (Rothe al.)
66.6
90.3
Non-auttorogreesive
Laser Lasernggeerr (Malmi et al.)
52.2
84.1
FELIX (Mallinson al.)
61.31
88.78
EDIT5 (Mallinson et al.)
64,95
Ours
66.0
90.7
Table 4: Sentenee fusion results on DiscoFuse Test data. BERT and RoBERTaaue are Seq2Seq models but initial- ized with 24-layer BERT and RoBERTa weights, respectively.
Speed Comparisons We compare different mod- els in terms of inference speed on CoNLL14 14 in the last column of Table 3. For fair comparisons, all of our models are run on a single Nvidia Tesla V $00 GPU with roughly 10,000 tokens per batch. We use BART-based Seq2Seq with decoding beam size of 12 as the speed benchmark. We incorporate the KV cache trick (Pope al., 2023) to climinate redundant computations. It takes about 45 sec- onds parse all 1.312 CoNLL14 sentences. As we can see, our CTC delivers a 4. x speedup against Seq2Seq and even faster than GECToR (2.9x), which also operates non-autoregreessivlly It owes much to the fact that CTC requires fewer iterations of refinement. SAD (Sun et al., 2021) achieves similar efficiency to ours but with much smaller (12+2) model size. Overall, our model performs orders of magnitude faster than Seq2Seq under similar conditions, readily meeting
the demands of online inference.
4.2 Sentence Fusion
Sentence fusion is the task of fusing several inde- pendent sentences into single coherent text.
Setup We train our sentence fusion models on the balanced Wikipedia portion of DiscoFuse data (Geva et al., 2019) following Mallinson 2022). For evaluation, we report two metrics (Geva et al., 2019), i.e., Exact Match (EM), which measures the percentage of exactly correct predic- tions, and SARI (Xu et al., 2016), which computes the averaged F1 scores of the inserted, kept, and deleted n-grams. For consistency, we use Geva et al. (2019)'s implementation' to compute SARI.
(2020,
can conclude that
Results are listed in Table 4. We can see that our model surpasses all non-autoregressive works sig- nificantly, especially EDIT5, by more than point
WWe observe that the KV cache trick brings significant spocdup. Our CTC model performs over 30x faster than the basic implementation without the trick.
3nttps://git.

---

BEA19
T
56
P
R
Fo.5
CTC (vanilla)
57.9
46.1
Fu.s (%)
CTC (ours)
61.8
43.4
57.0
52
GLAAT
60.2
44.2
2
60.9
43.8
56.5
48
60.9
44.2
56.6
60.7
45.1
56.8
10
20
30
40
50
Table 5: The results of vanilla CTC and proposed variant BEA19 Dev at stuge 1. "T"" stands for the upsampling ratio. GLAT" means removing the GLAT trick during training.
Training iterations
The training curves of vanilla CTC (red) and our CTC variant (blue), respectively. We plot the averaged Fo.s scores stage BEA Dev data each iteration, along with the upper/lower bound for different runs.
Figure 2:
EM score. One key observation is that 10.5% out of 4.5M fusion examples require source reorder- ing. LaserTagger (Malmi al., 2019) deals with this by defining a SWAP operation while EDITS uses pointer networks instead. The results indi- cate that our model capable of doing reordering implicitly and thus handles the fusion task skill- fully. On the other hand, our model achieves final EM/SARI scores of 66.0/90.7, showing strong com- petitiveness with the best performing RoBERTashare (66.6/90.3).
$ Why she couldn't recall anything after that ? </>>
Why didn't she
recall anything after that
(a) CTC (vamilla)
Why she
couldn't 't recall anything after that
Why couldn't she
recall anything after
(b) CTC (ours)
Figure 3: Two prodictions made vanilla CTC and CTC (ours), respectively. The connected lines produced align- ments. Green lines signify tokens directly copied from the source. The line thickness decided by the confidence of the prediction. The correct edit "she couldn't ' couldn' i she". while vanilla CTC wrongly corrects "couldn' 'f" "didn It".
5 Analysis
We demonstrate the superiority of our proposed CTC model by making comparisons from two per- spectives: 1) with vanilla CTC; 2) with other text- editing systems.
gain of 1.5 Fo.5 points.
5.1 Comparisons with Vanilla CTC
Alighnment behavior We give some prediction examples made by vanilla CTC and our proposed CTC variant in Fig. 3. We draw two observations from the figure: 1) in contrast to vanilla CTC, our proposed CTC variant copies most of the tokens in the prediction from the source text, thereby reduc- ing the over-correction phenomenon some extent and respecting the minimum edit principle of GEC (Ng et al., 201 4); 2) the predicted alignments of our proposed CTC variant are more in agreement with human opinions than those of vanilla CTC. We at- tribute the difference largely to the copy operation, which serves as pivot to guide the model on how to align with the source tokens, thereby allowing for more sensible edits.
Ablation We study the effectiveness of our pro- posed copy-aware CTC in Table 5. It is clear that our model brings remarkable gains over the vanilla CTC, especially terms precision, by points. This suggests that introducing the copy operation can effectively suppress the over-confident revi- sions of vanilla CTC, thus greatly reducing the errors. We also study the impact of the GLAT trick and upsampling ratios in the Table. We can con- clude that GLAT has nen-negl contribution (0.9) to the final results. Additionally, as the upsam- pling ratio T grows from 2 8, the results increase from 56.1 to 57.0 and later diminish; the optimal ratio was found to be 4.
Convergence behavior In Fig. 2, we plot the training curves regarding Fo.s scores and iterations of the two models. From the figure, we clearly see that it took about 10 iterations for our copy-aware CTC to reach the peak results, while 40 iterations for vanilla CTC. Our proposed CTC variant con- verges much faster than the vanilla one, with a final
5.2 Comparisons across Different Systems
We conduct series of comparisons here between our proposed CTC, GECToR (a representative Seq2Edit model), and BART-based Seq2Seq. gain a deeper understanding of the pros and cons of our model.

---

German
Russian
68
GEGTRRR BART 1CTC
P
R
Fo.5
P
R
Fo.5
Autoregressivve
62
Náplava et al.
78.21 59.94
73.71
63.26
27.50
50.20
Sun et al.
74.31
61.46
71.33
61.40
27.47
49.24
Fo Fo.a
gTs
75.96
51.62
56
mBART
53,50
26.35
44.36
mBART
73.97
53.98
68.86
32.13
4.99
15.38
50
mTSe
67.19
25.20
mTSuge
70.14
27.55
Non autoregressive
44
8-16
16-24
24-32
332
CTC 71.2
57.8
68.0
36.0
27.3
(12.75)
(27.73)
(22.71)
(17.49)
(19.32)
WER (%)
Table 6: Multilingual GEC results on German FaRKo-MERIIN Test data and Russian RULEC-GER Test data. Our model is trained on 24-layer XLM RoBERTa, which is similar in scale to mBART & mTS5ma (12+12), and is smaller than m'T5us (24+24).. means using synthetic data for pretrain- ing. mBART: Katsumata and Komachi (2020); gTS, mT5:
Figure 4: Fo.s scores braken down by word error rate (WER) on BEA19 Dev data. Numbers in parentheses represent the pereentage of gold edits cach group.
Rothe et al. (2021).
sertions required to edit the source sentence to the target, and dividing it by the number of tokens in the source. From the figure we can observe that the performance of BART is superior to CTC on sentences with low WERs, but tends to degrade WERs increase. BART performs worse than CTC by a large margin when WERs>-0244 indicating that BART-based Seq2Seq models tend to make rel- atively conservative edits. This fact is also implied in Table 3, where BART exhibits higher precisions and lower recall scores, resulting an inferior overall score than CTC. On the other hand, there is a large gap between GECToR and CTC for WERs> 0.32. showing that GECToR makes too stringent modifi- cations and can hardly surpass CTC without several rounds of refinements such scenarios.
Multilingual results To validate CTC can be well generalized to other languages, we conduct multilingual GEC experiments on German and Rus- sian, using their own portions of CLANG-8 data for training and Falko-MERLIN & RULEC-GEC Test data for evaluation, respectively. The results are presented in Table 6, where GECToR results absent as are aware of no GECToR extensions for these languages until now.3 We can see that our CTC performs similar to (m)BART on German and surpasses it by Fo.5 points on Russian. Further- more, CTC outperforms similar sized mase by 0.1 and 2.1 Fo.5 points on German and Russian, and is on par with m on Russian data, demon- strating the effectiveness of our method in multi- lingual settings. We highlight that, to the best of knowledge, we are the first non-autoregrrsssvve model that performs on par with Seq2Seq counter- parts on multilingual data. It is also worth noting that our models are purely trained on CLANG-8 and we do not pursue any data-augmentation tricks for further enhancements as it is beyond the scope of this work. We believe that our model can be greatly improved by introducing more synthetic data or increasing the model size, which we leave for future work.
Regarding flexibility It is known that the out- put flexibility of existing Seq2Edit models like GECToR is affected by the size of its small pre- defined vocabulary (Mallinson et al., 2022: Me- sham et al., 2023), but to what extent? We conduct some quantitative analyses in Table 7 to answer this question. Specifically, we measure how well the three systems can do when producing edits out- of GECToR vocabulary (Oov) by reporting their (token-level) ERRANT Fo.s results. The scores are categorized into three primary caase-grained error types, which correspond to the real edit operations in the GECToR vocabulary:
Regarding WERs We report Fo.s scores of the three systems broken down by word error rate (WER) in Figure 4. WERs are computed by count- ing the number of substitutions, delctions and in-
ORTH & NOUN represents back-and and-forth versions of lower/upper cases and singular/plural forms, e.g., - It] and [citizen-> citizens]. The associated GECToR operations involve only fixed string conversion rules and thus are not plagued by Oov problems.
'There are some pilot efforts to adapt GECToR to other languages, but with few exceptions (Zhang et al., 2022a). GECToR still hasn't I achieved comparable performance to Soq2Scq yet. See discussions in their code issues.

---

ORTH & NOUN
VERB
OTHER
der to provide hints for the decoder. There also broad strand of works that combine the autore- gressive fashion with edit operations (Stahlberg and Kumar, 2020; Reid and Zhong, 2021; Reid and Neubig, 2022, inter alia). Despite promising results, the slow inference speed limits their ap- plications in real-life online systems. To combat this, Sun et al. (2021) make use of very shallow decoders well aggressive copy strategy speed up the decoding. Chen et al. (2020) suggest to decoding spans needed to edit only for acceler- ation. Panthaplackel et al. (2021) present a novel dynamic programming algorithm to allow for span copy during generation. However, this does not break the inherent flaws. In this work, we instead propose a purely non-autoregressive approach for text editing, demonstrating great efficiency benefits over autoregressive counterparts.
Oov (%)
Total
Oov (%)
Total
Gold
(3.7)
(8.0)
GECToR
68.4
(0.0)
59.0
19.3
(2.2)
56.1
BART
67.9
27.1
(1.6)
56.3
42.0
(4.2)
58.5
CTC
67.9
24.8
(2.0)
58.3
39.1
(5.5)
59.5
Table 7: Fo.s scores of the three systems BEA19 Dev di- vided hy different edit types. The column "Oov (%)" contains Fo.s results of the corrections not covered by operations the GECToR vocabulary and their corresponding pereentages. The column "Total" contains overall Fo.s scores for the type.
VERB refers to verb form transformations like VB - VBZ (Imake-makes)) and VB - VBD (ldraw-> drew]), which are token-specific. GEC- ToR pre-defines a very large verb-form vocabu- lary handle these cases.
OTHER refers to other miscellaneous types that fall into the basic ADD, or REPLACE, operations in GECToR, e.g., REPLACE, ([The-aa).
From Table 7, we observe that it's hard for GEC- ToR to deal with the corrections not covered by the vocabulary.* Accordingly, the performance of GECToR is highly correlated with the proportion of Oovs in gold references. For the first two types, GECTOR provides very good coverage, with only 3.7% of gold VERB cases not included in the op- cration space. GECToR performs favorably better than BART and CTC in this scenario. However, for OTHER, there is a considerable portion (8.0% of gold references not contained in the vocabulary, and hence GECToR is greatly influenced. In con- trast, both BART and our CTC can produce more flexible corrections, with around 4.2% and 5.5% edits that can not be produced from the limited GECToR operation space, leading to great improve- ments against GECToR
Flexible text generation To promote more ef- ficient text editing, Malmi et al. (2019); Awasthi et al. (2019) tackle the task as a sequence tagging problem, predicting edit operations with a non- autoregressive decoder. Further developments have been made by GECToR (Omelianchuk et al., 2020), which enhances the method by designing many n- gram and language-dependen edit transformations. e.g., HYPHEN for combining separated tokens, and CAPITAL for capitalizing the first letter of the word, etc. However, the outputs of GECToR are arguably less flexible as its searching space is restricted in fix small-sized vocabulary. Further, it can hardly produce complex corrections with many insertions, reordering or rephrasing. For this reason, GECTOR heavily relies on several (typically 5) rounds of it- erative refinements to achieve good performance. FELIX and EDIT5 (Mallinson et al., 2020, 2022) present to use pointer networks and mask infilling decoder to aid this. But this inevitably incurs more model parameters and more decoding phases.
6 Related Works
Efficient text editing For years, Seq2Seq models have been hitherto the best approaches for text edit- ing (Vaswani et al., 2017; Lewis et al., 2020) due to their effectiveness. Some recent works obtain notable gains by injecting edit information into the generation process, making it more controllable and interpretable (Li al., 2022). SynGEC (Zhang et al., 2022b) integrates edit templates into syntax trees, which are then encoded with GNNs in or-
Compared with the above works, our proposed method capable of generating very flexible out- puts while enjoying similar inference efficiency to GECToR (see $ 5.2), which we attribute to the min- imal edit operation set in CTC as well as latent edit alignments. We hope our copy-aware CTC will serve as strong bascline for text editing to elicit further explorations. One possible direction is to improve it with stronger inter-token depen- dencies, e.g., tree structures (Gui al., 2023), to further reduce conditional independence (Huang et al., 2022b,a). we leave this to our future work.
"Strictly speaking, although difficult, it is possible for GECToR to make Oov corrections by multiple edit opera- tions. For example, one can mimic the Oov REP REPLACE by UpPERyple followed by PLURAL. That explains why the Oov Fo.s scores for GECToR are not necessarily 0.

---

7 Conclusion
References
In this work, we propose a novel CTC-based method for non-autoregressive text editing. The key idea is to subsume the editing process with latent CTC alignments. We further make cru- cial extension by introducing the copy operation, enabling very effective edits with three main oper- ations only: ADD, KEEP and DELETE. We conduct experiments on GEC and sentence fusion, showing that our method can reach or surpass the current state-of-the-art works while being 4x faster, and generalize well across German and Russian. More- over, compared with the best performing Seq2lddit model, our method also exhibits very good gener- ation flexibility, which could shed light on further studies on non-autoregressive text editing.
Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal. Sabyasachi Ghosh, and Vihari Piratla. 2019. Parallel iterative edit models for local sequence transduction. In Proceedings of EMNLP-IJCNLP.
Christopher Bryant, Mariano Felice. Øistein E. Ander- sen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical enror correction. In Proceedings of BEA, pages 52-75.
Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017. Automatic annotation and evaluation of error types for grammatical error correction. In Proceed- ings of ACL. pages 793-805.
William Chan, Chitwan Saharia, Geoffrey Hinton, Mo- hammad Norouzi, and Navdeep Jaitly. 2020. Imputer: Sequence modelling via imputation and dynamic pro- gramming. In Proceedings of ICML, pages 1403- 1413.
Limitations
Mengyun Chen, Tao Ge. Xingxing Zhang, Furu Wei, and Ming Zhou. 2020. Improving the efficiency of grammatical error correction with erroneous span detection and correction. In Proceedings of EMNLP. pages 7162-7169.
Decoding burdens We have noticed that the de- coding demands of our method are more substan- tial than GECToR when limited to a single round of iteration. This can be attributed to the upsam- pling step before decoding, which must expand the inputs to Tx the original length. Such burdens hinder our capacity to employ more precise yet time-intensive decoding strategies like prefix beam search (Maas al., 2014). We are going pursue a more efficient method that requires relatively small upsampling ratio.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better eval- uation for grammatical error correction. In Proceed- ings of NAACL, pages 568-572, Montréal, Canada.
Daniel Dahlmeier, Hwee Tou Ng, and Siew Mei Wu. 2013. Building large annotated corpus of learner English: The NUS corpus of learner English. Proceedings of BEA, pages 22-31.
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of NAACL-HLI, pages 4171 4186.
Precision-Recal tradeoff One potential limita- tion of our proposed copy-aware CTC is that it exhibits more pronounced over-correction behav- iors when compared GECToR and BART. This is reflected in the fact that our method tends to present higher recalls but relatively lower preci- sions, which may result in inferior results on some tasks and datasets with a greater emphasis on pre- cisions. We plan to explore strategies that result in better P/R tradeoff in the future.
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. EditNTS: An neural pmogrammer-intepreeee model for sentence simpli- fication through explicit editing. In Proceedings of ACL, pages 3393-3402.
Tao Fang, Shu Yang. Kaixin Lan, Derek F. Wong, Jin- peng Hu, Lidia S. Chao, and Yue Zhang. 2023. Is chatgpt highly fluent grammatical error correction system? a comprehensive evaluation.
Acknowledgements
We would like to thank the anonymous reviewers for their valuable comments, and Prof. Zhenghua Li for very helpful discussions. This work was supported by the National Natural Science Foun- dation of China (No.62076173), the High-level Entrepreneurship and Innovation Plan of Jiangsu Province (No.ISSCRC2011522) and the Project Funded by the Priority Academic Program Devel- opment of Jiangsu Higher Education Institutions.
Mor Geva, Eric Malmi, Idan Szpektor, and Jonathan Berant. 2019. DiscoFuse: A large-scale dataset for discourse-base sentence fusion. Proceedings of NAACL, pages 3443-3455, Minneapolis, Minnesota.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of ICML.

---

Jiatao Gu and Xiang Kong. 2021. Fully non- autoregressive neural machine translation: Tricks of the trade. In Findings of ACI-IJCNLP, pages 120- 133, Online.
Jonathan Mallinson, Jakub Adamek, Eric Malmi, Ali- aksei Severyn, and Alexander Fraser. 2022. Edit5; Semi-autoregressivee text editing with 15 warm-start. In Findings of EMNLP.
Shangtong Gui, Chenze Shao, Zhengrui Ma, Xishan Zhang, Yunji Chen, and Yang Feng. 2023. Non- autoregressive machine translation with probabilistic context-free grammar. In Advances in NIPS, New Orleans, USA.
Jonathan Mallinson, Aliaksei Severyn, Eric Malmi, and Guillermo Garrido. 2020. FELIX: Flexible text edit- ing through tagging and insertion. In Findings of EMNLP.
Eric Malmi, Yue Dong, Jonathan Mallinson, Aleksandr Chuklin, Jakub Adamek, Daniil Mirylenka, Felix Stahlberg, Sebastian Krause, Shankar Kumar, and Aliaksei Severyn. 2022. Text generation with text- editing models. In Proceedings of NAACI.
Awni Hammun. 2017. Sequence modeling with ctc. Dis- till
Fei Huang. Tianhua Tao. Hao Zhou, Lei Li, and Minlie Huang. 2022a. On the learning of non-autoregressive transformers. In Proceedings of ICML, pages 9356- 9376.
Eric Malmi. Sebastian Krause, Sascha Rothe. Daniil Mirylenka, and Aliaksei Severyn. 2019. Encode, tag. realize: High-precision text editing. In Proceedings of EMNLP-IJCNLP.
Fei Huang, Hao Zhou, Yang Liu, Hang Li, and Minlie Huang. 2022b. Directed acyclic transformer for non- autoregressive machine translation. In Proceedings of ICML, pages 9410-9428.
Stuart Mesham, Christopher Bryant, Marek Rei, and Zheng Yuan. 2023. An extended sequence tagging vocabulary for grammatical error correction. In Find- ings of EACL, pages 1608-1619, Dubrovnik, Croatia.
Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020. Encoder-decoder models benefit from pre-trained masked language models in grammatical error correction. In Proceed- ings of ACL, pages 4248 4254, Online. Association for Computational Linguistics.
Jakub Náplava and Milan Straka. 2019, Grammatical error correction in low-resource scenarios. In Pro- ceedings of WNUT, pages 346-356.
Courtney Napoles, Keisuke Sakaguchi, and Joel Tetreault. 2017. JFLEG: A fluency corpus and bench- mark for grammatical error correction. In Proceed- ings ACL, pages 229-234.
Satoru Katsumata and Mamoru Komachi. 2020. Stronger baselines for grammatical error correction using pretrained encoder-decoder model. In Pro- ceedings of AACI, pages 827-832.
Hwee Tou Ng. Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christo- pher Bryant. 2014. The CoNLL-2011 shared task on grammatical error correction. In Proceedings of CoNLL, pages 1-14.
Logan Lebanoff, Franck Demonccurt, Doo Soon Kim, Lidan Wang. Walter Chang, and Fei Liu. 2020. Learning to fuse sentences with transformers for sum- marization. In Proceedings of EMNLP, pages 4136- 4142, Online.
Kostiantyn Omelianchuk. Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. GECToR - grammatical eтror correction: Tag, not rewrite. In Proceedings BEA.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sqqeenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of ACL, pages 7871- 7880.
Kostiantyn Omelianchuk, Vipul Raheja, and Oleksandr Skurzhanskyi, 2021. Text Simplification by Tagging. In Proceedings of BEA, pages 11-25.
Sheena Panthaplackel, Miltiadis Allamanis, and Mare Copy that! editing sequences
Jiquan Li, Junliang Guo, Yongxin Zhu, Xin Sheng, De- qiang Jiang, Bo Ren, and Linli Xu. 2022. Sequence- to-action: Grammatical error correction with action guided sequence generation. In Proceedings of AAAI.
Brockschmidt. 2021. by copying spans. In Proceedings of AAAI, pages 13622-13630.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya. Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Efficiently scaling transformer in- ference. In Proceedings of MLSys.
Jindřích Libovický and Jindřích Helel. 2018. End-to- end non- oon-autoregressive neural machine translation with connectionist temporal classification. In Pro- ceedings of EMNLP, pages 3016-3021, Brussels, Bel- gium.
Lihua Qian. Hao Zhou, Yu Bao. Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. 2021. Glancing transformer for non-autoregressive neural machine translation. In Proceedings of ACL-IJCNLP. pages 1993-2003, Online.
Andrew I.. Maas, Awni Y. Hannun, Daniel Jurafsky, and Andrew Y. Ng. 2014. First-pass large vocabulary continuous speech recognition using bi-directional recurrent dnns.

---

Machel Reid and Graham Neubig. 2022. Learning to model editing processes. In Findings of the EMNLP. pages 3822-3832
#Sents
Error (%)
Usage
CLANG-8
2,372,119
57.8
Stage
FCE
34,490
62.6
Stage II
NUCLE
57,151
38.2
Stage II
Machel Reid and Victor Zhong. 2021. LEWIS: Leven- shtein editing for unsupervised text style transfer. In Findings of ACL-IJCNLP.
W&ILOCNNESS
34,308
66.3
Stage II III
BEAT9
4,384
65.2
Validation
CoNLL14
1,312
72.3
Test
Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebas- tian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of ACL-IJCNLP, pages 702-707, On- line.
Table Statistics of English GEC data. #Sents and Error (%) refers the number of sentences and the proportion of erroneous sentences, respectively.
Yue Zhang. Bo Zhang. Zhenghua Li, Zuyi Bao, Chen Li, and Min Zhang. 2022b. SynGEC: Syntax-enhanced grammatical error correction with a tailored GEC- oriented parser. In Proceedings of EMNLP, pages 2518-2531, Abu Dhabi, United Arab Emirates.
Sascha Rothe, Shashi Narayan, and Aliaksci Severyn. 2020. Leveraging Pre-trained Checkpoints for Se- quence Generation Tasks. TACL. pages 264-280.
Chitwan Saharia, William Chan, Saurabh Saxena, and
Mohammad Norouzi. 2020. non-autoregressive ma- chine translation with latent alignments. In Proceed- ings of EMNLP, pages 1098-1108, Online.
A Training Details
Table 8 gives the detailed statistics of English GEC data used for trainingvaaldaatinneesinngg the model. We train our models for at most 64 epochs based on roberta-large cess is terminated once the performance on Dev data does not improve after 10 epochs. The ex- periments are conducted GPU, and it took about 30 hours to finish the Stage GEC pretraining on CLANG-8 data. We train the models with roughly 100,000 tokens per mini- batch and use Adam W for model optimization with 31 = 0.9, 32 = 0.9, 10-12, The learning rate is set to 5 x 10 5 x 10 6 and 1 x 10 6 for stage ll & III, respectively. We also multiply the de- coder learning rate by factor of 10, and warmup the model by 1000 steps for better convergence. Regarding other tasks and languages, we gen- erally inherit the aforementioned hyperparameters with some adaptations. For multilingual German and Russian, we take x1m-roberte aateerralageee as the base model. Taking into account that the size of German and Russian data is relatively small, we set the batch size to 10,000 tokens and the lcarning rate 10-5 accordingly.
Chenze Shao, Zhengrui Ma, and Yang Feng. 2022. Viterbi decoding of directed acyclic transformer for non-autoregrressvve machine translation. In Findings of EMNLP, pages 4390-4397, Abu Dhabi, United Arab Emirates.
The training pro-
Felix Stahlberg and Shankar Kumar. 2020. Seq2Edits: Sequence transduction using span-level edit opera- tions. In Proceedings of EMNLP.
single Tesla A100
Xin Sun, Tao Ge. Shuming Ma. Jingjing Li, Furu Wei, and Houfeng Wang. 2022. A unified strategy for multilingual grammatical error correction with pre- trained cross-lingual language model. In Proceed- ings of IJCAI, pages 4367-4374.
Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. 2021. Instantancous grammatical error correction with shal- low aggressive decoding. In Proceedings of ACL. pages 5937-5947, Online. Association for Computa- tional Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in NIPS, pages 5998-6008.
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Hallison-Buuchh 2016. Optimizing statisti- cal machine translation for text simplification. TACL, pages 401-415.
Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. new dataset and method for automatically grading ESOL texts. In Proceedings of ACI, pages 180-189.
Yue Zhang, Zhenghua Li. Zuyi Bao, Jiacheng Li. Bo Zhang, Chen Li, Fei Huang, and Min Zhang. 2022a. MuCGEC: a multi-reference multi-source evaluation dataset for Chinese grammatical error cor- rection. In Proceedings of NAACL.


Reviews:
[{"review_id": "zEOz0reLbL", "reviewer": "Reviewer_L81y", "summary": "", "questions": "A. line 072: A clarification question: the criticism of the existing Seq2Edit models states that these models rely on linguistic features to customize the edit actions. It remains unclear for me if this is meant in a general way (needing to establish the pattern of house : houses, chair : chairs as meaningful) or if the method explicitly relies on POS tags. Could you clarify this?\n\nB. Line 204: How was the value of T=4 decided? \n\nC. Line 412: “the predicted alignments of our proposed CTC variant are more in agreement with human opinions than those of vanilla CTC.”  This is a strong statement. What is it based on?\n\nD. Table 6: when comparing your method to pre-existing models, are you rerunning the evaluation for the pre-existing models or reporting the results from pre-existing papers? If you are rerunning the evaluations, why are there missing P and R scores for gT5 and mT5? If you are reporting results that were published before, it should be stated clearly and the papers correctly cited. This stands for all evaluations.\n\nE. Line 470: I found the section “Regarding flexibility” very hard to follow. Most comments seem unnecessarily convoluted. E.g., “We observe that GECToR performs awkward in OOV cases, but for the first two types, GECToR has very good coverage with only 3.7% gold VERB references not covered. GECToR performs favorably better than BART and CTC in this situation.”. Does this mean the following: “GECToR has extremely bad performance on OOV VERB edits (0 F0.5 score). However, there is a very low proporition of OOV VERB edits in the reference corpus. Therefore, GECToR’s performance on VERB edits is not overly affected by this shortcoming.”? Could you clarify the rest of the comments? I am interested in your results here but am not sure I am understanding the text correctly.", "limitations": "", "rating": 4, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer L81y,\n\nThank you for your detailed and helpful comments.\nWe are committed to improving the quality of our paper based on your suggestions.\nAddressing the typos and enhancing the clarity of the writing are key aspects that we will focus on in the next version.\n\n> It remains unclear for me if this is meant in a general way (needing to establish the pattern of house : houses, chair : chairs as meaningful) or if the method explicitly relies on POS tags. Could you clarify this?\n\nWe would like to clarify that the representative Seq2Edit-based GECToR method encompasses both scenarios you mentioned.\nThe tagger introduces `SINGULAR`/`PLURAL` tag to handle the singular and plural conversion of nouns, `LOWER`/`UPPER` to handle the case conversions, etc.\nIn more intricate cases, GECToR leverages the power of POS tags to make nuanced distinctions.\nFor instance, it employs the `VB-VBZ` tag transformation to convert verbs like \"make\" to \"makes\", and similarly employs `VBZ-VBG` to transform words like \"thinks\" to \"thinking\".\n\n> How was the value of T=4 decided?\n\nIn our preliminary experiments, we have tried different values of `T`, ranging from 2 to 8, as detailed in Table 5.\nWe identified that setting `T=4` yields the best performance on BEA19 data without sacrificing too much speed.\nSo we directly fix the value of `T` to 4 in the latter experiments.\nWe will clarify this in the next version. Thank you!\n\n> “the predicted alignments of our proposed CTC variant are more in agreement with human opinions than those of vanilla CTC.” This is a strong statement. What is it based on?\n\nConsidering the examples in Fig. 3, we can see that when empowered by the copy mechanism, our CTC variant demonstrates the capacity to properly generate the alignment for phrases like \"couldn't\" and \"couldn't she\".\nThis achievement mirrors the human decision-making process for alignment, which entails identifying common tokens between source and target sentences, and using them as reference points for alignment.\nWe posit that the direct copying capability in our model reduces over-correction tendencies, ultimately leading to enhanced results.\nWe will enhance the clarity of our explanations in the subsequent edition. Thank you for raising this concern.\n\n> when comparing your method to pre-existing models, are you rerunning the evaluation for the pre-existing models or reporting the results from pre-existing papers? If you are rerunning the evaluations, why are there missing P and R scores for gT5 and mT5? If you are reporting results that were published before, it should be stated clearly and the papers correctly cited. This stands for all evaluations.\n\nFor our comparison with pre-existing models, we have sourced the results directly from the original papers without re-running the evaluations.\nIn Table 7, we have referenced the papers for mBART, gT5, and mT5 in a succinct manner due to space constraints.\nWe are committed to making these citations more prominent and clearly explaining our approach in the upcoming version. Thank you for bringing this to our attention.\n\n> I found the section “Regarding flexibility” very hard to follow. Most comments seem unnecessarily convoluted. E.g., “We observe that GECToR performs awkward in OOV cases, but for the first two types, GECToR has very good coverage with only 3.7% gold VERB references not covered. GECToR performs favorably better than BART and CTC in this situation.”. Does this mean the following: “GECToR has extremely bad performance on OOV VERB edits (0 F0.5 score). However, there is a very low proporition of OOV VERB edits in the reference corpus. Therefore, GECToR’s performance on VERB edits is not overly affected by this shortcoming.”? Could you clarify the rest of the comments? I am interested in your results here but am not sure I am understanding the text correctly.\n\nThank you for your kind clarification.\nIn Table 7, the OOV column presents a comparison among the three methods based on the prediction scores for instances where token-level GECToR edits are unable to be applied.\nIt is noteworthy that, despite the challenges, GECToR does have the potential to generate OOV predictions through the combination of multiple edits, as evidenced by the example in Footnote 4.\nOne crucial aspect we wish to underscore is that GECToR is severely restricted by the limited vocabulary space compared with CTC and BART.\nConsequently, the overall performance of GECToR is strongly influenced by the proportion of OOV cases.\nNotably, the most prominent prevalence of OOVs emerges in the `OTHER` category (8\\% of gold references), causing GECToR to exhibit its weakest overall performance (56.1) in this specific category.\nIn contrast, both CTC and BART exhibit a higher capacity to generate adaptable outputs, thereby yielding higher F0.5 scores.\nWe will revise our writing to make it more clear.\n\n> Consider clarifying why in ASR CTC assumes that the input is longer than the output, and why that is not the case in text generation.\n\nIn ASR, the input consists of a sequence of acoustic units, naturally exceeding the length of the output token sequence.\nOn the contrary, in text generation tasks such as English-to-German translation, both input and output are sequences of tokens, and the former is not inherently longer than the latter.\nTo fulfill the assumptions underlying the CTC methodology, we indeed employ an upsampling technique to magnify the input sequence.\nThis aligns with the requirement that the input surpasses the output in terms of length.\nWe promise to provide a clearer and more detailed explanation in the next version, thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**NB**\n\nI would like to specify that I was recruited as an emergency reviewer and that the topic of the paper is not in my domain of expertise. Therefore I have a hard time evaluating the novelty and the impact of the method beyond what is presented in the paper. This will be reflected in my reviewer confidence score.\n\n----------------------------------------\n\nIn general, this seems like a well thought out work. The claims are supported by a wide range of experiments, and the paper provides a lot of information on how it was all done. In particular:\n\n1. This paper presents an approach that is novel and effective, leading to interesting improvements both in terms of results and computational performance. An additional positive aspect is that the method is based on adapting an approach from a different NLP domain (ASR). The adaptation is straightforward and clearly described.\n2. Solid evaluation of the proposed method. The method is evaluated on two different tasks, against multiple existing models. Several targeted evaluations are done in order to examine different aspects of model performance.\n3. Generalizability to other languages. The authors state explicitly that their goal is to create a method that is language-independent and can be applied to other languages. They test this by applying their model to Russian and German with satisfying results.\n4. The authors indicate that their code is available on github and say that the model will be released.", "reasons_to_reject": "I have not identified any major issues that would warrant rejection. However, as stated above, this is not my domain of expertise. I do have a number of smaller issues to raise, which will be done in the questions and typo/style suggestions.", "questions_to_authors": "A. line 072: A clarification question: the criticism of the existing Seq2Edit models states that these models rely on linguistic features to customize the edit actions. It remains unclear for me if this is meant in a general way (needing to establish the pattern of house : houses, chair : chairs as meaningful) or if the method explicitly relies on POS tags. Could you clarify this?\n\nB. Line 204: How was the value of T=4 decided? \n\nC. Line 412: “the predicted alignments of our proposed CTC variant are more in agreement with human opinions than those of vanilla CTC.”  This is a strong statement. What is it based on?\n\nD. Table 6: when comparing your method to pre-existing models, are you rerunning the evaluation for the pre-existing models or reporting the results from pre-existing papers? If you are rerunning the evaluations, why are there missing P and R scores for gT5 and mT5? If you are reporting results that were published before, it should be stated clearly and the papers correctly cited. This stands for all evaluations.\n\nE. Line 470: I found the section “Regarding flexibility” very hard to follow. Most comments seem unnecessarily convoluted. E.g., “We observe that GECToR performs awkward in OOV cases, but for the first two types, GECToR has very good coverage with only 3.7% gold VERB references not covered. GECToR performs favorably better than BART and CTC in this situation.”. Does this mean the following: “GECToR has extremely bad performance on OOV VERB edits (0 F0.5 score). However, there is a very low proporition of OOV VERB edits in the reference corpus. Therefore, GECToR’s performance on VERB edits is not overly affected by this shortcoming.”? Could you clarify the rest of the comments? I am interested in your results here but am not sure I am understanding the text correctly.", "paper_topic_and_main_contributions": "This paper presents a novel approach to text editing. The proposed method is based on a Seq2Edit approach, which relies on applying a sequence of edit operations over the input, as opposed to the previous generation of methods based on a classical Seq2Seq approach. The advantage of the Seq2Edit approach is that it is much faster than Seq2Seq. The main contribution of this work is applying CTC (connectionist temporal classification) to this task. CTC is in general used in ASR. The method proposed here interprets the CTC alignments as a series of edit operations that can then be executed over the input sequence. Crucially, the authors introduce a “KEEP” edit  operation (in addition to the traditional “ADD” and “DELETE”), which denotes that an input token can be directly copied to the output. The authors credit this modification with the important speed gains of their method. Special attention is given to output flexibility and generalizability to other languages.\n\nThe method is evaluated on two text edit tasks: grammatical error correction and sentence fusion. It is evaluated against a wide range of existing methods. In both cases, the proposed method is on par or better than the reported existing methods. Additionally, on the grammatical error correction task, the proposed method is 30x faster than the Seq2Seq model at inference time, and there is also evidence that the modified CTC approach converges faster at training time than the standard CTC approach. The authors also investigate the relationship between model performance and WER, as well as the output flexibility. They test the multilingual generalization capacities of the method by applying it to German and Russian with encouraging results.\n\n-----------------------------------------\n\n__*Post-rebuttal comment*__  \nThe authors have addressed my concerns in a thorough and thoughtful manner. Their answers strengthen my impression that this is a solid and well thought out contribution and will make a good paper once the suggestions are integrated. I am keeping my grades unchanged because my concerns were minor and the grades were based on my perception of the scope and the contribution of the paper rather than the concerns themselves.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "In general, the paper is very dense and occasionally hard to follow. While I salute the richness of the information provided, the paper may benefit from a more careful selection of the content. It would make it easier to read. In any case, I strongly suggest a major editing pass focused on making sure that the main ideas and concepts are clearly presented and easy to grasp.\n\nAbstract: CTC, GEC – consider spelling these out in the abstract. The first occurrences in the paper body are too far to be useful.\n\nSection 5: it would be useful to state explictly on which task the comparisons in Section 5.1 are done. This can be deduced from the table and figure captions, but it makes the reading more difficult.\n\nThe dataset sizes are not given in the paper body. For the GEC task, they are given in the appendix,  but they are not given at all for the sentence fusion task. I would suggest making this information available in the paper itself: readers who are not familiar with the datasets would appreciate it. \n\nRelated Work: Consider moving this section after the introduction. If the introduction and the related work were closer together, it would allow you to shorten both of them, since some of the information is present in both.\n\nConsider expanding all contractions such as “it’s” (e.g. line 103)\n\nTable1: in the GEC example, in the target sentence, should the first ‘to’ also be in blue? \n\n061: meanwhile orders of magnitude faster → while being orders of magnitude faster.\n\n064: there arise several challenges → consider replacing by ‘several challenges arise’\n\n112: four-folds → four-fold\n\n115: to our best knowledge → to the best of our knowledge\n\n120: favorably better → better\n\n149: factorize → factorizes\n\n164: from the regressive counterpart → from its regressive counterpart\n\nFigure 1 is not referenced in the text of the paper.\n\nLine 171: Consider clarifying why in ASR CTC assumes that the input is longer than the output;, and why that is not the case in text generation. \n\n173: resulting → resulting in\n\n180: grants the power of CTC → grants power to CTC?\n\n190: give → given\n\n196: once getting the hidden states → once the hidden states are obtained\n\n216: that → of\n\n221: explicit model → an explicit model\n\n222: We in this work propose → In this work, we propose\n\n230: likelyhood → likelihood\n\n252: parallely → parallelly\n\n253: Table. 2 → Table 2\n\n277: representive → representative?\n\n307: which has huge impact on final results → which has a huge impact on the final results\n\n310: for better access of the results → for more comparable results\n\n319: our models → our model\n\n315: is superior to BART on all datasets by 0.4, 0.1, and 2.2 scores → remove ‘scores’. Same at line 326\n\n345: that → than\n\n363: 1 EM score → 1 EM point\n\n384: precisions → precision\n\n399: while → with\n\n401: with about 1.5 final F0.5 gains → with a final gain of about 1.5 F0.5 points\n\n403: Alighnment → alignment\n\n434, 436: scores → points\n\n464: resulting a inferior → resulting in an inferior\n\n472: are → is\n\nTable 7 needs a clearer caption. The explanation in the text is hard to follow too.\n\n506: related works → related work\n\n566: we leave this as our future works → We leave this to our future work.\n\n577: current state of the art works with over 30x times faster → current state of the art while being 30x faster\n\n582: shed lights → shed light"}}, {"review_id": "TXEElDcJV9", "reviewer": "Reviewer_M58B", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer M58B,\n\nThank you for your very constructive feedback.\nWe will try our best to improve the writing and polish the paper according to your comments in the next version.\n\n> Insufficient exposition of the Gamma function implementation, at least a few sentences should be allocated to explain how this is implemented\n\nThank you for highlighting this aspect.\nIn Table 2, we give a toy example for the sentence ***I like an dog***, wherein ***K K K K $\\emptyset$ $\\emptyset$ $\\emptyset$ dogs*** represents one of the possible sequences returned by the Gamma function.\nTo enhance clarity, we recognize the need to offer a more comprehensive explanation of the implementation. \nWe will add more examples to mitigate any potential ambiguities in the upcoming version.\n\n> lower precision than GECToR (but somewhat better recall results in overall better F1), large expansion of the input length makes decoding slower\n\nRegarding your observation about precision compared to GECToR, we acknowledge this trade-off and would like to emphasize that our model's strength lies in its superior F0.5 scores across various tasks and languages. \nThis underlines its effectiveness in a broader range of contexts.\nWe also wish to draw special attention to the inherent advantage of our model in generating more flexible outputs when compared to GECToR (see Section 5.2).\n\nFurthermore, we'd like to clarify that despite the challenges posed by the upsampling layer, our proposed model maintains a commendable speed even when compared to GECToR.\nNotably, GECToR also operates in a non-autoregressive manner, and our method's efficiency stems from the significantly reduced number of refinement iterations (2 v.s. GECToR's 5).\nWe acknowledge the importance of efficient decoding methods and recognize that future research could delve into strategies that accommodate inputs of shorter lengths while preserving performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel method of using CTC to model GEC with NAR decoding\n- Clear exposition and relatively thorough evaluation\n", "reasons_to_reject": "- Insufficient exposition of the Gamma function implementation, at least a few sentences should be allocated to explain how this is implemented\n- lower precision than GECTOR (but somewhat better recall results in overall better F1)\n- large expansion of the input length makes decoding slower ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors present a non-autoregressive (NAR) text editing algorithm which extends CTC with a copy operation, and models editing with latent alignments from CTC. NAR output with flexible length is enable by 4x upsampling input length. The proposed approach extends upon recent work in the seq2edit paradigm, where a target sequence is generated from a source sequence by applying a series of edit operations. Latent edit operations are marginalized over during training to maximize target sequence likelihood. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "this paper has many typos, needs careful proofreading\na few of them:\n030-034 - Table 1 does not show text summarization (it shows GEC and sentence fusion)\n040 fasion → fashion\n230 log-likelyhood → log-likelihood\n251 predicts → predict\n252 parallely → in parallel\n"}}, {"review_id": "odQ8O6zK8c", "reviewer": "Reviewer_BNoe", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer BNoe,\n\nThank you for your insightful feedback.\n\n> Although this work claims to be the first to adapt CTC for text editing, it is worth noting that other recent works have also explored related non-autoregressive text generation concepts. In my opinion, this work appears to amalgamate pre-existing techniques rather than introducing significant technical novelty.\n\nWe appreciate your consideration of related concepts in non-autoregressive text generation. \nWe want to emphasize that while there have been some explorations in non-autoregreesive machine translation (NAT), our work focuses specifically on adapting CTC for text editing, which presents unique challenges in compared to NAT due to the inherent overlap between inputs and outputs in such tasks.\nTypicial Seq2Edit methods like GECToR address this by employing a bunch of elaborate tags to make cautious edits.\nIn contrast, vanilla CTC, unlike both GECToR and our proposed method, lacks an effective mechanism to effectively tackle these challenges, resulting in suboptimal performance.\n\nOne of the primary contributions of our work is the incorporation of a copy mechanism into the CTC framework to address the complexities of text-editing tasks, which involves modifying the learning objective to effectively handle the alignment of copy-aware paths.\nWe achieve this by providing a novel dynamic programming algorithm, which marginalizes all possible copy-aware alignment paths during training.\n\nWe are pleased to direct your attention to Fig. 2 and Table 5, where we provide a visual representation and quantitative analysis of our proposed method's effectiveness. \nWe show that our approach not only yields enhanced convergence behavior but also achieves an overall improvement of 1.9 F0.5 scores when compared to vanilla CTC on BEA19 GEC data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It exhibits good cross-lingual generalization capabilities by performing competitively to Seq2Seq models on German and Russian GEC with a single model.\n\n2. The paper demonstrates robust experimental results on GEC and sentence fusion tasks, outperforming previous non-autoregressive methods.", "reasons_to_reject": "Although this work claims to be the first to adapt CTC for text editing, it is worth noting that other recent works have also explored related non-autoregressive text generation concepts. In my opinion, this work appears to amalgamate pre-existing techniques rather than introducing significant technical novelty.", "questions_to_authors": "", "paper_topic_and_main_contributions": "\nThis paper proposes and evaluates a novel method for efficient non-autoregressive text editing based on adapting connectionist temporal classification (CTC) to incorporate copy actions. Compared to GECToR, the proposed method has better flexibility and language generalization. The main contributions include: (1) Adapting CTC by introducing copy actions like KEEP, ALLOWING direct modeling of edit operations flexibly; (2) Demonstrating strong performance surpassing prior Seq2Edit models on standard GEC and sentence fusion datasets; (3) Exhibiting cross-lingual generalization capabilities by achieving comparable performance to Seq2Seq on German and Russian GEC; (4) Revealing through analysis the robustness of the proposed model across various scenarios and its ability to generate more flexible outputs.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a non-autoregressive seq2edit approach for text editing in order to improve inference speed. The novel approach is based on connectionist temporal classification, traditionally used in ASR. The method is evaluated in two tasks: grammatical error correction and sentence fusion.

Reviewers agree that the method is effective, clearly described and straightforward to apply. There is also a solid evaluation procedure, comparing the approach against multiple existing models. Evaluation is also cardied out in other two languages other than English with satisfying results. Finally, the paper also includes a through analysis of the robustness of the proposed model across various scenarios and its ability to generate more flexible outputs.

A main concern is related to the novelty of the approach, since similar ones have been proposed for machine translation. The authors have clarified how their apporach differns, and are encouraged to highlight these differences in the paper. Some concerns were raised regarding details of certain model components, which were clarified during rebuttal. There is also a concern that the paper is dense, and some suggestions were provided for improving the clarity. Authors are encouraged to include these recommendations and clarifications in the final version.
Decision: Accept-Main