Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose an efficient insertion-based text generator (INSNET) that does not require re-encoding of the context. The model follows the insight to enable computation sharing and aggregation. The proposed model is also flexible for the controlled generation and parallel decoding. Experiments on two unsupervised lexically constrained text generation and machine translation tasks verify the model. 
strengths_and_weaknesses: strengths:

1. The proposed model is versatile and intriguing, which contributes to general text generation models. It also achieves a better quality-latency trade-off compared to former edit-based generations. 
2. The novel and well-motivated scheme for position representation and insertion slot representation, which is insightful for the text generation community.
3. The efficient computation sharing strategy is well-illustrated. The adaptive training scheme is easy to understand as it follows the curriculum learning to start from generic decoding to arbitrary insertion orders. Though it brings certain overhead, experiments show the decoder stays efficient. 

weaknesses:

1. Relatively lack of experiments for analysis. 
2. The impact of hyper-parameter $\tau$ is task-specific and may be costly to tune. The paper should provide some quantitative analysis, e.g., why machine translation can achieve decent quality with smaller $\tau$.
questions: Is it possible to conduct an ablation study with only position representation or slot representation? How much impact does each of these elements contribute? 
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This article proposed INSNET, an insertion-based text generation model, for efficient training and flexible decoding. Specifically, a new insertion-oriented position encoding method is used in INSNET to enable efficient computation and a hyper-parameter is provided in INSNET to flexibly control the parallel degree. Experiments were performed to demonstrate INSNET’S advantages in terms of training speed, inference efficiency, and generation quality.
strengths_and_weaknesses: Strengths:

This article proposed a new insertion-based text generation model, which is efficient in time consumption and flexibly supports parallel and sequential decoding.

Weaknesses:

There lack ablation experiments to verify the effect of three major components of INSNET.

It is unclear what useful information is captured by one pass of context encoding in INSNET.


questions: 1. What is the precondition to use this position encoding method, this method seems heavily rely on device support (eg GPU support on matrix operation). Under the precondition, what extra cost would it cause considering it is time-efficient, is it memory efficient compared to other methods?

2. From what Figure 3 illustrated, it seems that the algorithm actually recomputes position encoding in each step, thinking that in each time step, the sequence position is overall recomputed throw matrix operation. In fact, there is no need to recompute previous token position encoding. So, is there any way to go further to avoid this computational waste and maintain efficiency in the same way?

3. t is an important hyper-parameter for INSNET since it can control how much to parallel. It makes sure the flexibility of the model. Experiments should be made to demonstrate that truly influence the parallelism of the model. What is the relationship between and Sequential Ratio?  Are there any formulas that could be given to show how it manipulates the parallel capability of the model?

4. The inference time comparison could also be analyzed to show the model’s efficiency. More analysis about inference time comparison should be given.

5. Compared to the traditional transformer, the training speed is slower, why happen? There should be some reason to be listed in the paper and verified? 

6. Since the non-autoregressive transformer is not that common, when introducing INSNET, a figure about it could be given. More explanation could be given to it.

7. Figure 6 shows the performance of INSNET under different Sequential Ratio, could the same experiments be executed for another model to show the overall performance comparison among these models under different Sequential Ratio?

8. The definition of the Loss function contains an error. In the second part of the formula, the symbol l is from 0 to m, while in the first part, l is between 0 and m-1. They are not consistent.
limitations: N/A
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes InsNet, an insertainn-based text generation model. It introduces a new insertion-oriented relative position schema for efficient encoding and computation. It introduced an algorithm to adaptively parallelize the generation process to speed up the inference. The paper conducts comprehensive experiments on a few text generation tasks, and the result is competitive. 

strengths_and_weaknesses: The paper introduced “offset”, an insertion-oriented relative position encoding schema. It overcomes the problem of prior methods and reduces the re-encoding problem. It works well in practice. The paper also offers an algorithm which decomposes this method into a few fast matrix operations. It’s a novel algorithm which can be widely applied on future insertion based models. 

The paper rigorously and closely compares with prior work. The proposed method works well on most datasets. The results are convincing. 

Weakness
The paper is very hard to follow. I can’t quite understand Sec 3.2 and 3.3, after spending 1+ hours reading this paper and some of the prior work (InsT and InDIGO). There are many concepts and terminologies in this paper that I can’t fully understand. eg. Dinic’s algorithm, fully-sequential layerization, tau, etc. I even had a hard time to understand the offset algorithm. I am not sure how the model works as a whole system (input, output, modules, training, inference, etc.) I was very excited about the paper, but I am frustrated after spending 2 hours. 
I would give some suggestions:
1. Use a simpler example. “
 I have a pen . 
” is too long for demonstrating the algorithm. It’s such a pain to read Figure 2, 3, and 4. I believe a sequence like ABCD is visually better and easier to remember.
2. Elaborate the core equations. There are a good number of equations in Section 3, but it requires decent patience and memory to encode, understand and remember them. Please focus on core/your equations, and illustrate them with figures and proper notation. 
3. Move something to appendix. The paper gets pretty crowded and complicated. You can also consider just get rid of some components. 
4. Show some example outputs. I am curious about how it empirically works and what’s the order of generation. 

questions: What’s KD?
I can't fully understand Sec 3.2 and 3.3.
limitations: Poor writing and presentation.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new insertion-based model, called INSNET, that supports both sequential and parallel decoding, to tackle the inefficient training of the naive insertion-based model. They experimented on two un-supervised lexically constrained text generation datasets and three machine translation datasets, showing that the INSNET achieves better performance in each task. The proposed approach also greatly reduces the training cost against the previous approach (as reported in Table 4).


Typos, or minor revisions for readability:

- at l. 46: "encodings , i.e." -> "encodings, i.e."

- In Table 3, the citation format will be "Gu et al." -> "(Gu et al.)" and so on

- Please re-arrange Figures/Tables display in each page. e.g.  Figure 6 could be moved to the later page so that the description will show up on the same page


+++++++++
Thank you for addressing my questions! I also checked the uploaded pdf, and I confirm the corresponding descriptions that the authors added accordingly. Yet, the manuscripts will need to be fixed a bit more for the camera-ready, so I will keep my score as "Weak Accept".
strengths_and_weaknesses: - They conducted experiments on multiple translation tasks, experimentally showing the effectiveness of the proposed approach. In Table 3, they should also report the performance using KD in other method. This seems unfair comparison. Yet, the proposed approach successfully improves the training efficiency.

- Easy to apply the proposed approach to the existing models, by introducing one pass of transformer encoding computation.


questions: - Data set up for non-autoregressive MT tasks are not found. You need to add them into Appendix accordingly. 

- What kind of the error examples did you get in the experiments? Can you report any patterns if exist, or the errors that be specifically caused by the insertion-based approach?
limitations: N/A
review_title: null

Meta-review: metareview: There is consensus between 3/4 reviewers regarding acceptance and the review that suggests borderline rejection also admits the merit of the paper.  Given the new proposed method for insertion based generation in transformers and the experimental results, I think the paper should be accepted to Neurips.
confidence: Certain
Decision: Accept