Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new problem of modelling the series editing operations of an article, i.e. $p(x_1 \dots x_n )=\prod_t p(x_t | x_1\dots x_{t-1}) $, where each $x_t$ represents an article after t-th editing. There are three types of operations: insert, delete, and replacement. The paper propose a model based on a modified Transformer to predict both the edit operation and the content to be generated for insertion and replacement. To predict the edit operation type (including additional no-edit action), the model uses a Transformer encoder to encoder the k previous version of history (if $k=1$, it is just based previous version) and a simple autoregressive MLP to predict the type of editing operation for each of the token. For consecutive insertion and replacement edits, the model will take the average contextual representation of the span to generate tokens (together with the span start and end index and edit type embedding).
For evaluation, the paper constructs two datasets with editing history: WikiRevisions (from wikipedia editing history with some cleaning) and CodeRevisions (from 700 python repositories on Github with MIT license, over 1000 commits and 500 stars).
The paper propose a set of three metrics: operation perplexity, generation perplexity (given operation), and the editing perplexity combining both. On these metrics, the paper shows the proposed EditPro model improves over baseline LEWIS method on WikiRevisions data. It also shows more edit history is helpful in prediction, therefore EditPro based on three previous revision history is better than that based on one immediate previous version. 

The paper also propose three downstream tasks for evaluation: editing given the text comment (from the commit message), edit-conditioned generation, and edit-condition classification. 

main_review: Strong points of the paper:
1. Previous approaches only consider predicting editing based on the current text (therefore just one revision history). This paper considers more revision to predict and it show more revisions (up to three) are beneficial. 
2. The paper prepares two corpus with full revision history, which might be beneficial for the community on future research. 
3. The paper propose a modified semi-autoregressive Transformer model to predict both the editing operator and generator for insertion and replacement operations. The model is reasonable. 
4. The paper shows improvement on the metrics on WikiRevisions dataset. 

Weak points of the paper:
1. One of the metrics proposed in the paper is slightly strange. Why to measure the perplexity of operation? Why not measuring the accuracy or F1 score of the per-token (excluding the no-edit operation). 
2. There are some details missing about the method in Section 3.2. What is the network for autoregressive operation prediction $P(e_j | e_1^{j-1})$? What is the decoder for insertion and replacement? Is it Transformer decoder? 
3. Examples in Table 4 actually show that this EditPro model generates rather bad text. Those text are fluent but semantically incoherent. 
4. continue from the above point, Human evaluation of the editing quality is missing. Do you conduct human evaluation of the generated text? 
5. It seems a rather simple baseline is missing. If you directly use Transformer encoder-decoder to predict both the operation and the generation text, how is EditPro compare to this baseline?

It could be a good paper if the authors could fix some major concerns above. 

Some minor issue: 
It seems the perplexity equations are incorrect, missing negative sign. 

There are also additional related work on editing based method for text generation. [1] uses simulated annealing to search for edit operations, and [2] uses Metropolis-Hastings sampling for editing. Do those apply to the edit generation problem here? 

[1]: Unsupervised Text Generation by Learning from Search. 2020.

[2]: CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling. 2019
summary_of_the_review: Reason to accept: new problem setup for editing based text generation, two new datasets, and evidence of multiple history helps generation. 

Reasons to reject: one metric is questionable, examples showing the method not working, missing human evaluation and a straightforward baseline. Without those, it is hard to judge the quality of the generation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new task of modeling editing processes to model the whole process of iteratively generating sequences. To tackle the new challenge, the authors propose a conceptual framework to describe the likelihood of multi-step edits and describe neural models that can learn a generative model of sequences based on these multi-step edits. The Experimental results show that modeling editing processes improve performance compared to previous single-step models of edits on related downstream tasks and the proposed task.
main_review: Strengths:
1. The idea is simple and intuitive
2. The paper is well organized and easy to follow

Weaknesses:
1. My main concern is the motivation of this work. It is not clear why interactively editing is better than single-step editing. The authors need to elaborate on the source of the gain in more detail.
2. Similar idea (interactively editing) is already presented in work[1]. The authors do not take this method as a baseline, or at least they should thoroughly discuss the differences with this work.
3. In section 6, the paper only compares their model to a limited baseline, making the experiment less convincing. The paper needs to add some baseline, especially in the dataset CODEREVISIONS.

References: [1] Shaohan, Yu, Furu, Ming (2018). Text Morphing. 
summary_of_the_review: Please refer to the Weaknesses in Main Review, I hope the authors could give a clear answer
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the task of explicitly modeling the process of editing sequences in an iterative manner, while the prior approaches for sequence editing are usually single-step editing methods. Thus, the proposed method is significantly novel. The proposal is to decompose the editing process into (i) predicting edit operations (insert, substitute, delete, and keep) and (ii) generating spans corresponding to those operations. Both these steps are conditioned on the previous versions of the input sequence and its edit history. The model first predicts edit operations conditioned on previous documents and then revises the document conditioned on edits and previous documents.

In my opinion, the experimental section can be improved significantly. It currently lacks sufficient comparisons with prior work to demonstrate the efficacy of iterative edit modeling over prior methods of single-step editing. The proposed evaluation metrics also seem to be directly aligned with the design of the proposed model, hence comparisons with other models on these metrics might be unfair.
main_review: Strengths:
 - Modeling the task of sequence editing as a multistep, iterative process. Almost all previous methods model sequence editing as a single-step process.  The proposed method is technically sound.

 - Edit processes are modeled on a larger scale e.g. document-level edits. 

 - Modeling multi-order edits up to the order of 3, does result in improved performance (but only in comparison to the same model with an edit order of 1)  

 - New proposals for datasets and metrics related to iterative sequence editing: Edit Perplexity, Generation Perplexity, and Operation Perplexity. (However, the model's design explicitly favors these metrics.)

Limitations / Concerns
 
- Comparisons with single-step edit models: There are prior works on iterative sequence editing that utilize single-step edit models and find them useful. E.g. see Table 4 in [1]. How effective is the proposed model in comparison to these methods?

 - The paper starts with the motivation that "Revising and editing are a central part of the human creative workflow, with most original content being developed not in a single iteration, but in many iterations with each more refined than the last". I agree with the authors on this part. However, the datasets considered in this paper may not be well aligned with this motivation. E.g. in many cases, Wikipedia edits and code edits may not be just stylistic edits required for improving sentence formation. A good portion of edits might also introduce new information as well. 

 - Grammatical Error Correction is a very well-known task in the domain of sequence editing and aligned with the motivation of iteratively refining and editing a given sentence. The experimental observations could have been more convincing if the authors could show that their method is more effective than prior single-step edit methods for such tasks.

 - The proposed evaluation metrics in Section 5.2, are very much tied to the model design (Eqn 6). Thus the proposed metrics may naturally favor the model over other models which were not trained using similar objectives. Hence, I feel that the proposed model requires a more rigorous evaluation. 

 - The proposed method is only compared with a single baseline. In the related work, authors do acknowledge prior work on sequence editing. However, a comparison with those methods is currently not provided.

 - Details about the posterior regularization being used in Section 6.2 are not fully clear.

 - It would be interesting to know how important is the order of edit modeling, in Table 2 and Table 3? Does performance gains diminish beyond order 3?


References:

[1] Parallel Iterative Edit Models for Local Sequence Transduction (https://aclanthology.org/D19-1435.pdf) 


summary_of_the_review: This paper investigates an interesting problem of modeling the iterative process of editing the sequences. The ideas proposed seem to be technically sound and novel.

However, I am not convinced with the tasks, metrics, and comparisons with prior work on sequence editing. 

Metrics: In my opinion, metrics seem to be directly aligned with the model's design, and hence comparing with models which do not optimize these metrics could be unfair.

Tasks: Tasks like Grammatical Error Correction or paraphrasing are widely known and directly aligned with the paper's motivation of iteratively editing. The effectiveness of the proposed method on such tasks can make experiments significantly more convincing.

Comparisons with prior work: The paper compares with only one baseline, while it does acknowledge well-known and recent papers in the domain of sequence editing. The paper rightly argues that most of the prior work models sequence editing as a single-step process. However, the significance of modeling edits in multiple iterative steps will be more evident after comparisons with single-step edit models.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Existing edit-based models condition on a single sentence or document. This paper argues that such frameworks do not represent processes well in which the same document undergoes multiple iterations of revisions, for example the edit histories of Wikipedia articles or source code. The proposed system is a two-stage model that first labels each token in the next revision with one of four edit labels, and then employs a seq2seq second stage model to predict the output tokens for generative labels such as INSERT and REPLACE. Both stages do not only condition on the previous revision but on the n latest revisions. Gains in perplexity are shown, as well as improvements of two downstream tasks: "edit-conditioned generation" (generating meta information to edits such as comments) and "edit-conditioned classification" (1-way semantic intent classification).
main_review: The outset of the paper is valid: Conditioning on the complete (or longer) revision history can potentially model the editing process of documents more accurately than models that just take the latest history into account. However, I'm not completely convinced by the evaluation. I think that it is hard to justify an approach with perplexity improvements only. Perplexity can be a valuable tool to get a better understanding of the system, but ultimately showing the usefulness in a downstream task can make a much stronger case. The two downstream tasks in the paper that do not use perplexity (Tab.3 / Sec. 6.2) are (a) predicting code commit messages / Wikipedia edit comments and (b) semantic intent classification. If I understand correctly, both of these tasks actually do not require the *prediction* of the document edits. Tab. 3 does not compare with related work on these tasks. BLEU and F1 scores do improve with increasing order which is a good sign, but simple baselines/ablations would strengthen the results. For example, task (a) may simply benefit from having access to previous commit messages, not from a better modeling of the edits. Similarly, for (b) the model could learn that two subsequent "Vandalism" revisions are unlikely without using edits from previous revisions.

Minor comments:
- The paper sometimes claims that previous models are just able to represent single *edits* (abstract, Sec. 4). This is, of course, not true: they are able to represent multiple edits with a document, just in a single *edit step*. This subtle difference is quite important so the wording should be more precise.
- Eq. 2: This might be true in theory, but it is highly impractical since it seems unlikely that the sum over all document revision histories can be approximated to a reasonable degree, let alone enumerated. 
- Eq. 2: The notation under the sigma is wrong. It should start with \tilde{X}\in\{...\}
- Sec. 2: "x_0 represents [...] generally the null string". Is this true for both WikiRevisions and CodeRevisions tasks? Doesn't this significantly increase the length of the edit representations for subsequent revisions? x_0 is not empty in Fig. 1.

summary_of_the_review: While there are some potential merits to the idea of modeling multiple revisions, I have doubts whether the evaluation demonstrates them in a convincing way.
review_title: null

Meta-review: This paper introduces a novel task (i.e., modelling the iterative process of editing sequences) and proposes a Transformer-based architecture to address it tractably. The paper also elects a number of metrics that are argued to shed enough light onto the merits of the proposed architecture. 

In our view, the current version is not ready for acceptance. Here are some of the reasons I'd highlight: 

* It is not entirely clear to us that the task in consideration has enough substance to grant acceptance nor that it speaks to a large enough audience. Perhaps the challenges identified here are more general and the developments for this task can be extended to related generation problems? If so, this is something one could consider for a revised version of the paper. 
* The motivation does not seem to align well with the datasets used to demonstrate the task. Perhaps the difficulty to find a dataset that matches the motivation is an indication that the task and its challenges are tad too specific. 
* It's the impression of more or less everyone involved that the paper lacks comparisons, and that the evaluation is not thorough enough, and the rebuttal did not ease our concerns sufficiently.
Decision: Reject