Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new decoding mechanism that allows  span-copying which can be viewed as a generalisation of
pointer networks.

Because action sequences of the same length can lead to sequences of different lengths decoding becomes tricky therefore authors propose a variation of standard beam search that calculates a lower bound of sequence probabilities rather than the true probability of generation this is achieved in practice by merging probabilities of sampled rays of actions generating the same sequence during the search.

one advantage of this proposed model is that it doesn't need to copy word by word to update sequences which need minor changes, rather than the seq2seq model with copy actions which due to the way we train those models using NLL loss will likely assign high probabilities to the non-modified input. 

Authors evaluate their model against traditional seq2seq models with copy actions over a set of tasks:
* code correction tasks: two bug-fix pair (BFP) datasets of Tufano et al. (2019)
* grammar error correction (Bryant et al., 2017)
* learning edit representations 

Pros: 
Overall I am in favour of this work acceptance it represents a neat modelling for copying sequences that integrated simply with seq2seq models, especially the transformer model. 

Cons: 
- One of the drawbacks of this method is the decoding strategy although authors present a motivated solution for that. The proposed variation of beam search by calculating the lower bound solution seems adhoc and some corner cases are not explained in the paper (see the question below). 
- Experiments could have been more thorough, especially in terms of architectures. I was disappointed not to see authors only comparing between GRU based seq2seq with copy actions, one baseline and their model implementation over a biGRU seq2seq. 

Questions to authors: 
- During inference using the proposed variation of beam search (e.g. k=5), What will happen for example if one ray of actions was dropped because not of the top 5 this ray of actions if continued using future actions would map to one existing top-scoring rays? do you do a way to control that? 
- What is the reason behind choosing the bi-gru architecture? 

Missing references: 
There are a couple of similar work that authors might want to add: 
* Latent Predictor Networks for Code Generation https://arxiv.org/pdf/1603.06744.pdf
* An Operation Network for Abstractive Sentence Compression https://www.aclweb.org/anthology/C18-1091.pdf
* Levenshtein Transformer https://arxiv.org/pdf/1905.11006.pdf
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper study the problem of editing sequences, such as natural language or code source, by copying large spans of the original sequence. A simple baseline solution to this problem is to learn a sequence to sequence neural network, which generates the edited sequence conditioned on the original one. This method can be improved by adding a copying mechanism, based on pointer networks, to copy tokens from the input. However, most of such existing approaches can only copy one input token at a time, which is a limitation when most of the input should be copied, which is the case for most editing tasks. In this paper, the authors propose a mechanism that can copy entire spans of the input instead of just individual tokens. In that case, a particular sequence can often be generated by many different actions (eg. copying individual tokens, pairs of tokens, or the whole span). It is thus important to marginalize over all the actions that generated a particular sequence. This can be done efficiently, using dynamic programming, if the probability of an action depends on the generated tokens only, but not on the sequence of actions used to generate them. In the case of neural network, this means that the decoder of the model takes the tokens as input, instead of the spans. To represent spans, the authors propose to use the concatenation of the hidden states corresponding to the beginning and end of the span. Then the probability of copying a span is obtained by taking the dot product between this representation and the current hidden state of the decoder, and applying the softmax. The authors evaluate the proposed approach on the following tasks: code repair, grammar error correction and edit representations (on wikipedia and c# code).

The paper is well written, and easy to follow, even if some sections could be a bit more detailed (for example, the section on 
beam search decoding). The problem studied in the paper - copying spans from the input - is interesting, and has applications in NLP or code generation. I think that the the proposed solution is technically sound.
However, I have some concerns regarding the paper. First I believe that many relevant prior works are not discussed in the paper, making some technical contributions of the paper not novel. For example, previous methods were proposed to copy 
spans from the input [1], to edit existing sequences [2], or to marginalize over different generation sequences
by conditioning only on the generated tokens (instead of the actions the generated the sequence) [3,4]. The body of work on iterative refinement for sequence generation is also probably relevant to this paper [5,6]. Additionally, I found the experimental section a bit weak, as most of the baseline used to compare seem a bit weak. The proposed method is mostly compared on datasets that are relatively new, or on tasks such as the grammar error correction where strong methods were excluded.

Overall, I found the paper well	written, and the proposed method to make sense. Unfortunately, I believe that the work is a bit incremental, most of the technical contributions having already been published. Since the experimental results are not very strong, I do not think the paper is good enough for publication to the ICLR conference.

== References ==

[1] Sequential Copying Networks, Qingyu Zhou, Nan Yang, Furu Wei, Ming Zhou, AAAI 2018.
[2] QuickEdit: Editing text & translations via simple delete actions, David Grangier, Michael Auli, 2017
[3] Latent Predictor Networks for Code Generation, Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, Andrew Senior, Fumin Wang, Phil Blunsom, ACL 2016
[4] Training Hybrid Language Models by Marginalizing over Segmentations, Edouard Grave, Sainbayar Sukhbaatar, Piotr Bojanowski, Armand Joulin, ACL 2019
[5] Mask-Predict: Parallel Decoding of Conditional Masked Language Models, Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer, EMNLP 2019
[6] Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement, EMNLP 2018
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: In this work, the authors tackle the problem of span-based copying in sequence-based neural models. In particular, they extend the standard copying techniques of  (Vinyals et. al., Gulcehre et. al., etc.) which only allow for single-token copy actions. Their span-based copy mechanism allows for multiple tokens to be copied at a time during decoding via a recursive formulation that defines the output sequence distribution as a marginal over the complete set of action combinations that result in the sequence being produced. The authors also propose a span-based beam decoding algorithm that scores output sequences via a sum over the probabilities of action sequences that produce the same output. 

The authors evaluate their model on four tasks: code repair, grammar error correction, editing wikipedia, and editing code. They find that their proposed technique consistently outperforms single-token copy-based seq2seq baselines. They also show that the efficacy of their proposed beam decoding mechanism and do some simple quantitative analysis that the model learns to copy spans longer than a single token.


In general, I found this paper to be very clearly written with very good motivation for the proposed solution. In addition, I thought the authors did a good job of testing their model against a wide range of benchmark problems. It seems that their copy extension is a meaningful contribution. 

I do, however, some questions regarding the evaluation, in particular the complexity of the baselines that were compared against. For example, the model consistently outperforms simple copy seq2seq baselines as well as the baselines in which the benchmark datasets were proposed (Tufano et. al, Yin et. al.) However, it does not seem the span-based copying method is state-of-the-art. If it is not state-of-the-art, how far off the SOTA is this proposed architecture? Did the authors do any analysis whereby the span-copy mechanism was added to an existing SOTA model, and if so, did this still produce gains? It's a bit difficult to situate the exact power of this new mechanism, given that it is often only compared to a simplistic copy-seq2seq method. 

Other questions/feedback I have for the authors:
1) How efficient/scalable is the proposed mechanism? I would like to see a more formal treatment of the run-time of the training marginalization operation.
2) It would be nice to see a quantitative analysis for distribution of sequence lengths copied over (like some sort of histogram) for the datasets.
3) It would also be helpful to add some short descriptions of the benchmark datasets.


review_title: Official Blind Review #3

Meta-review: This paper proposes an addition to seq2seq models to allow the model to copy spans of tokens of arbitrary length in one step. The authors argue that this method is useful in editing applications where long spans of the output sequence will be exact copies of the input. Reviewers agreed that the problem is interesting and the solution technically sound. However, during the discussion phase there were concerns that the method was too incremental to warrant publication at ICLR. The work would be strengthened with a more thorough discussion of related work and additional experiments comparing with the relevant baselines as suggested by Reviewer 2.
Decision: Reject