InsNet: An Efficient, Flexible, and Performant Insertion-based Text Generation Model
Authors: Sidi Lu
Tao Meng
Nanyun Peng
Venue: NeurIPS 2022 Accept (2022)
Keywords: text generation
insertion-based
neural machine translation

Abstract: We propose InsNet, an expressive insertion-based text generator with efficient training and flexible decoding (parallel or sequential). Unlike most existing insertion-based text generation works that require re-encoding of the (decoding) context after each insertion operation and thus are inefficient to train, InsNet only requires one pass of context encoding for the entire insertion sequence during training by using a novel insertion-oriented position encoding to enable computation sharing. Furthermore, InsNet provides a controllable switch between parallel and sequential decoding, making it flexible to handle more parallelizable tasks such as machine translation to support efficient decoding, or less parallelizable tasks such as lexically constrained text generation to guarantee high-quality outputs. Experiments on two unsupervised lexically constrained text generation datasets and three machine translation datasets demonstrate InsNet’s advantages over previous insertion-based methods in terms of training speed, inference efficiency, and generation quality.

INSNET: An Efficient, Flexible, and Performant Insertion-based Text Generation Model
Sidi Lu, Tao Meng, Nanyun Peng
University of California, Los Angeles
(sidilu, tmeng, violetpeng ) [cs ucla. edu
Abstract
We propose INSNET, an expressive insertion-based text generator with efficient training and flexible decoding (parallel or sequential). Unlike most existing insertion-based text generation works that require re-encoding of the context after each insertion operation and thus are inefficient to train, INSNET only requires one pass of context encoding for the entire sequence during training by introducing a novel -nsertion-oriented position encoding and weighted slot representation strategy to enable computation sharing. Furthermore, we propose an algorithm INSNET-Dinic to better determine the parallelization of insertion operations that provides a controllable switch between parallel and sequential decoding, making it flexible to handle more parallelizable tasks such machine translation with efficient decoding, or less parallelizable tasks such as open-domain text generation guarantee high-quality outputs. Experiments two lexically constrained text generation datasets and three machine translation datasets demonstrate IN- SNET's advantages previous insertion-based methods in terms of training speed, inference efficiency, and generation quality.
Introduction
Insertion-based text generation that formulates the generation process sequence of token insertion operations has received increasing attention in recent years. There are two major advantages of insertion-based generation over the prevalent left-to-right auto-regressive paradigm: 1) It reduces the decoding cost to sub-linear w.r.. the sequence length with parallel decoding (Stern et al. 2019 Gu
al. 2019b) and the flexible insertion orders may better recover/utilize the underlying linguistic
structures of languages (Welleck et al. 2019 Gu et al. 2019a).
However, this paradigm of text generation brings unique challenges, mostly in the training efficiency. Unlike left-to-right auto-regressive decoders which monotonically expand the context, the insertion operations complicate the position information of each token the context expands. Concretely, as is shown in Figure]] [ along with the insertion operations. As result, a naive implementation of insertion-based models (e.g., Sterrn et al. (2019); (): Gu et al. (2019b)) needs to re-encode the context with updated positional information for each token the insertions proceed, yielding inefficient training with O(n) times of context re-encoding (with n indicating the sequence length).
the absolute position of token in sequence constantly changes
To tackle this problem, previous insertion- based generation models such Insertion Transformer (InsT) (Stern et al.) 2019) and Levenshtein Transformer (Lev')) (Gu et al., (019b) propose parallel token insertion reduce the ceartiionrreennoddin steps from O ((n) to Glog n. for both training and inference. However, while it works well for machine translation, such parallel insertion falls short
on high-entropy generation tasks such as open-domain dialogue systems(Li et al.
[017aa. creative
generation such as stories (Yao et al. 2019 Goldfarb-Tarraan et al. 20200 Han et al. 2022), poetry
(Manurung al. 2000 Tian and Peng 2022). and humor generation (Hempelmann 2008 He et
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

RRe-Iine
dRee BRe-Enn.
Step
poss
2
Model
w/ ParaDee
w/ SeqDee
PadIfo
Trans.
e(logn) m)
O(s)
Absolute
Lev. Trans.
(0gg
Qin)
Absalute
Step t+1
pas
pex 1
pas 2
pre- pes 3
pos 4
pare par
NMSTG
Ofa]
O(n)
Markovian/Absoluee
InDIGO
NBA
0(1
Direction only
Figure I: The absolute position information of each token is volatile for insertion- based models. Thus, naive models with absolute position encoding have to re-encode the se- quence after each insertion operation during training.
INSNET (Ours)
O(1)
O(1)
Relative
Table 1: Comparisons between INSNET and existing insertion-basee models regard- ing training-time re-encoding steps (#Re-Enc. columns) and how the models encode posi- tional information (the Postnfo column).
2019; Mittal et al. 2022 Tian et al. 2022; Sun et al. 2022 where the output is open open-ended and the tokens usually highly dependent each other, thus cannot be inserted in parallel. Another thread of work seeks to maintain sequential insertions but simplify the position encoding for the tokens so that they do not change as the insertion operations proceed (Gu et al., 2019a) to save computation. However, this sacrifices model capability and limits its applications to complex sequences.
In this paper, we propose INSNET, which addresses the training efficiency issue of insertion- based text generators by enabling computation sharing similar as that in vanilla decoder transformers. Our first contribution is
relative positional encoding coined offset that allows INSNET to achieve computation sharing without compromising model capacity. At each insertion step, previously computed position encodings of the existing tokens remain unchanged, while the position encodings of newly inserted tokens accurately recording the updated pairwise token spatial relations of all the inserted tokens. A corresponding offset can be efficiently computed for any given insertion order with a novel process named offset compression. In this way, we avoid expensive re-encoding of the context, and based on the encoded context, our second contribution is the design of an effective aggregation strategy that allows us to parallelly generate expressive slot representations for every slot in the partially generated sequence, flexibly support both sequential and parallel decoding. Finally, inspired by the layerization idea of Dinic's 's algorithm(Dinizz (2066), we propose an algorithm for INSNET better determine the parallelization of insertion operations in order to reduce the likelihood discrepancy before and after the parallelization. With all the components, INSNET is a novel framework that enables efficient training, flexible decoding, and expressive positional encoding. Table I shows a comparison between INSNET and all prior insertion beaeed models.
2 Related Works and Background
Auto-regressive Language Models minimize the -eeative-log-likeliiee of a sequence of
tokens 8kn = Zo. Ii,..., .... In- with a left-to-right factorization. With the transformer architecture (Vaswani et al. 2017), each step of likelihood estimation can be calculated parallel while sharing the prefix context encoding calculations. This makes it possible to build powerful and efficient text generation models, like the GPT family (Radford et al., 2018 2019 Brown al. 2020). A lot of successful applications are based on this paradigm of models, such as automatic story generation (Yao) (ct al. 2019 Tan et al.) 2020), image captioning (Vinyals et al., 2015 Xu et al.) translation (Bahdanau et al. 2015 Liu et al., 2020), and dialogue system (Li et al.
2015), machine 2017b a).
Insertion-based Models with Parallel Decoding Insertion transformer (Stern al.
2019) (InsT)
proposes a design for insertion based text generation. In each step, a bi-directional encoder trans- former is performed on the partial sequence to compute the representation for each candidate slot between every two consecutive positions. Then, model for the joint distribution of position-toke is built to insert one or more token(s). Variants of InsT share the common atomic objective that models the step log-likelihood. On step & where a token Tajs+1 is inserted in slot 41i+1 between position i and + of context s. the step likeliool can be written as:
log p(1 44541/s7) log Pposition (i + 1|s?) + log Ptoken (Fussil(s)))))) @ c(s)))+1);
where e(-); stands for the i-th position of bi-directional encoding of the sequence and ( stands for vector concatenation. InsT adopts the original absolute positional encoding of transformers, the

---

representation of the generated sequence has to be completely re-encoded after each step of context expansion to match the position changes of tokens. The expectation of the negative log step likelihood over all permitted context-insertion pairs at each step is computed as the step loss. The step losses from the first step to the last one are summed up as the sequence loss. Benefiting from the partially parallelized prediction of tokens, Ins'' can reduce the number of re-encoding steps to (log n).
Levenshtein Transformer (Gu al. 2019b) (LevT) contains two phases during generation: 1) Insertion phase: it first uses a similar strategy as InsT, but only inserts placeholders; an MLM is applied to fulfiil the placeholders. 2) Deletion phase: the model is trained as token-wise discriminator to determine where to delete, under the evaluation of the Levenshtein distance by dynamic programming. In practice. this would result in slower process compared to InsT, but better generation quality as alleviates the incoherence caused by parallel insertions.
Sequential Insertion-based Model Previous exploration in insertion-based models mostly assumed the usage of parallel decoding in each decoding step, resulting in a partially auto-regressive procedure. For those are trained to only generate one token per decoding step, a vanilla implementation would result in an (nn) factor in training time complexity. NMSTG (Non-monotonic Sequential Text Generation, Welleck al. 2019 is one of the first attempts at modeling a non-monotonic sequential insertion-based generation process. It constrains the dependencies of each inserted token to pseudo- Markovian on an expansion tree. InDIGO et al. 2019a) is proposed as a sequentially--ecodee insertion bbasti model with the transformer architecture. It supports the efficient likelihood estimation by working around the aforementioned volatility problem (see Figure ! at a cost of omitting the distance information in its position encoding, and is thus able to adopt the conventional computation sharing trick to boost the multiplicative factor in training time complexity for each sequence to as fast as (1). However, InDIGO uses a encoding of all previously inserted tokens to predict the next token regardless of their tentative position. It is only after the next token is predicted, the inserted position for it is predicted. Thus, there's 's no trivial solution to use InDIGO as a (partially) non-autoregressive insertion-based generator.
Absolute vs. Relative Position Embedding The original transformer (Vaswani et al. 2017) uses sinusoidal, absolute position embeddings. Relative position embedding in transformers (Shaw et al.) 2018; Dai et al. 2019) was originally proposed to make the modeling of spatial relation invariant to position translation, and improve the long-term dependency performance of the model. In replacement of absolute positions, which are tied to each token in the sequence, relative positions try to encode the spatial layout of the tokens with a matrix that records a directed distance from the column token to the row token. We further develop the idea of relative position embedding as the key component to overcome the issue of volatile position information of absolute positions in an insertion- generation process. A noticeable fact is that InDIGO's implementation of position encoding can also be regarded as a direction-only relative position embedding system.
3
The INSNET Model
There are three major components of INSNET: 1) An context encoder based on the transformer architecture (Vaswani et al. 2017) that uses a novel way to compute invertion-oriented relative position encoding to better suit the insertion-based nature of the generation process and enable computation sharing (Section 3.1); 2) module to compose expressive slot representation for predicting tokens to insert in different slots simultaneously (Section 3.2); and 3) an algorithm that adaptively determines the parallelization of the insertion operations to minimize conflicts (Section 3.3 Figure illustrate the components of the full model.
3.1 Context Encoder: A Transformer with Insertion-Oriented Relative Position
In the introduction, we discussed the challenges in (efficient) computation sharing caused by the volatile position information of the context tokens as shown in Figure [ We address this problem by empowering the transformer-basee pairwise relative position encoding. The proposed insertion-oriented relative position encoding shares important designs with previous relative position embeddings(Da et al., 2019: Yang et al. 2019 Shih et al.) 2019), but differs in how it accurately depicts the spatial layout (i.e., the actual sequential order) of the inserted tokens in insertion-bait basedd generation process.
context encoder with distance-aware, insertion-one insertion-oriented
3

---

Encoder
Inputs
[See S85-2.1
Posiicon Inforation jAS ondesklngs
Full Susaance
1
Order Perrusation ids
1
9
anpetlonceeeneed postinn recodincl
Insenton Operidon Sieuu eccuence
InsNet-
Brary Locaos
Pikk ESNDOO
Step Comesuulized
Siop Strpul
bn Food isserting C dic glaes Top dki
Teken Ensseddingg
DE
Uniinsed hozering givmn slye
Projection and Re combination
Slot Representation
Stop Lodl
right glaba
Sanp
Skep 2.8
globl
Slop
Step
Step
Sinp Sky Stap 3.6
Hige
Stap2
Sap
Stap3
glibal
Stpz Sue OD
Stap
Toom Mellnodd
Step 1 represarcations
E Eostreed
Tuken
Bosthoed
Full Enplie Orjectie Trsiiing
Figure 2: Illustration of the full InsNet model. Given input sequence with specified insertion order as illustrated the [Inputs] panel (the upper right panel blue), a context encoder (Section (.1] illustrated in the [Encoder] panel (the upper left corner in purple) with insertion-oriented position coding (Section 3.1.I is applied to the sequence of insertion operations to obtain step-contcxtualized token embeddings as the transformer outputs. Then, the slot representation module (Section 3.2 il- lustrated in the [Slot Representation] panel (the bottom panel yellow) compose slot representations for each step from InsNet transformer outputs using its left, right token representation and global, step-wise token representation. The slot representations are then used to compute the token/position likelihood and also to determine the auto-pat cattoopraallllzztiinn in InsNet-Dinie (Section 3.3) illustrated in the [InsNet-Dinic] panel (the middle right panel in green).
3.1.1 Offset: Insertion-Orientee Relative Position Encoding
To illustrate our relative position encoding, considering the partial context (to be completed by further insertions) of "I have pen" with an insertion order of "have pen 1". To insert an "a" between "have" " and "pen", the distance vector for token "a" against the rest of the context inserted far should be [("have", -I)) ("pen", +1), ("T", -2)], simplified as [-1, +1, -2]. This relative position encoding clearly defines where the insertion happens by only describing the pairwise spatial relationship between the incoming token and the existing context tokens. We can pack the relative position vectors for each insertion step get matrix that reflects the relative spatial relation along the trajectory of insertions, with each row corresponding to an insertion step. We name the offset matrix.
<p djphavvepe
CP ciphaveeen
<g>
have
pen
</>>
Step
6G
-5
U
1
1
2
have
haav
Step
pen
prn
Step
34
12
-1
3
3
J
Step
Step
56
2
:5
-3
-4
11
(a) Absolute Positions
(b) Vanilla Relative Positions
(c) Our Relative Positions
(d) Ours, Order Restored
Figure Comparison of different position encodings for an insertion-based generation process.
From left to right, we illustrate 3(a) the volatile absolute positions, 3(b) the traditional non-insertion- oriented relative positions, 3[c) the proposed offset matrix presented the insertion order, and (3d) the same offset matrix permuted to show how it looks like if we restore the actual )sequence.

---

5
Permutation indices
Step
lile
Into
N
6
4
5
0
inf
inf
inf
ind
inf
inf
0
Step
Step
Strp 4:
replace
6
S
upper
0
2
inf
ind
ind
Inf
elempents
2
diagonal element
4
triagular
with
ind
ind
int
2
inf
inf
low-to-high rank
each row
infinitty
6
S
+
2
inf
Figure 4: Illustration of the offset compression algorithm. efficiently transforms the insertion order/permutation indices to an offset matrix.
As concrete example. to generate the sentence "I have pen." with insertion order "(BOS) (EOS) have pen I a .", the complete offset matrix is shown in Figure ((c) The effec- tive elements of this matrix computation sharing trick for the decoder transformers trivial. This reduces the number of context -eeceenooiing steps from n in naive insertion- based models (Ster et al. 2019) to I during training.
all in the lower triangular region, making the application of the
Discussion: Comparisons With Other Positional Encoding Strategies To illustrate how the proposed insertion-oriented relative position encoding is different from existing position encoding strategies, Figure 3 shows the volatile, un-reusable absolute positions in (3(a the traditional non- insertion-oriented relative positions as used in models such XLNer Yang et al. (2019) in 3(b) the proposed offset matrix in the insertion order in B(c) and the same offset matrix permuted to show the restored actual Seartial)sseueene
in 3(d) We can see that the absolute and the traditional relative generators. The former positional encoding strategies are not applicable for efficient insertion-base
requires constant updates of the positional encoding of the existing tokens as the insertion progresses, disabling computation sharing. The latter requires knowledge about the final sequence length at the beginning of the insertion process, which is unrealistic and destroys the flexibility of ineettionbbsse models. Our relative position encoding, on the other hand, reflects the order of the original sequence. Tokens that are on the left of the current one in the original sequence have negative relative position to the current one while preserving the insertion order. As is illustrated in Figure 3(d) all the "later inserted tokens" are masked and excluded from the previously inserted token's computation.
3.1.2 Efficient Computation of Offset Matrix
During training, given insertion order generate sequence, can pre-compute the offset matrix. However, naively constructing the offset matrix with element-wise operations takes O(n)) complexity for each sequence, limiting the training efficiency. Therefore we design an efficient algorithm to construct the offset matrix. Assume the insertion order is described in the form of absolute position permutation indices (i.e., the previous example. [0. 6, 2, 4, 3. 5)), we design matrix algorithm called offseet compression for our purpose. Figure 4 illustrates the algorithm with our exemplar sentence. Specifically, we first convert the absolute position vector into a matrix by duplication. Then, the upper triangular elements are masked by "infinity" to remove their impact in relative position computation because the inserting token should not attend to future to-be-inserted tokens. In the third step, each element is replaced by its in-row ranking i.e. its absolute position skipping the masked positions. In the final step, each row is baselined by the diagonal element to reflect that the relative position is between the last inserted token (the diagonal element) and previously inserted ones. The algorithm can efficiently executed as a series of fast matrix operations.
3.1.3 An INSNET Layer
We hereby describe how to incorporate the offset matrix into each transformer layer in INSNET for efficient context encoding. In general, we inherit most designs from previous ones with relative position encoding (Dai et al., 2019 Yang et al. 2019). The encoder panel in Figure 2 (the upper left panel in purple) illustrate the process. Specifically, for layer number i = I...N, given the output embedding E from the last layer and the offset matrix R, the formulation of transformer layer in an N laaer INSNET can be written as:
5

---

Q'K', V P W W W W
A K K + -
K' + v'
Vi
Paaseed-Softan Coftmax(A')V"
reduced
V reduced + =
Ei-!
skipconn
E
Feed-Forward V
skipconn 0²)
where Q¹ K' V V are query, key, value, position matrices, respectively, for layer i. u', v W W E W W and 0* are learnable model parameters of each INSNET layer. We denote the input word embeddings as E for notation consistency. The last layer of transformer in INSNET outputs the wise context-aware token embeddings. They will be used to compute slot representations for the insertion steps.
3.2 Slot Representation and Insertion Prediction
With the context efficiently encoded, the subsequent step of an insertion- based generator is predict the next position and token to insert. Thus, a representation for each potential insertion slot should be computed. We hereby show how slot representations can be aggregated from INSNET outputs and the potential challenges during this aggregation process.
A naive design of the slot representation is to simply concatenate the representation vectors from the left-neighbor and right- neighbor (in the natural observation order) of the slot as is done in prior works (Stern et al. 2019). This slot representation can efficiently compute the slot representation for all possible slots in parallel for each time step. However, the context encoding we obtained is insertion-order sensitive and unaware of the tokens inserted later in INSNET. This naive slot representation thus falls short of capturing the global sequence information. As a remedy, we propose (t) to also include the last insertion token's representation to compose the slot representation e Specifically, given the representations of the left-side neighbor e;., the right-side neighbor e4. and the last inserted token ex, the slot representation can be computed as:
LayeerNom((f((ee)) f-(e+))) + ee)
where fi and fr are linear projections for left-point and right-point representation vectors and @ stands for vector concatenation. The slot representation panel in Figure (the bottom panel in yellow) illustrates the process. Note that comparing step and step 4, both compute the slot representation for J. 
, but the resulting slot representation changes in response new step due the introduction of the global vector.
The slot representation can then be converted into probabilistic distribution over the vocabulary to predict the log-likelihood of inserted tokens using log-linear transformation i.e. log p(
$4j (
(8)
log-softmax (W e + b)_()))
(t) To decide which slot to insert next, a position logit ax w e lie
p [i
computed for each slot with a linear layer. Then a (log-)soft-man operation is applied on top of the
position logits to obtain the log-probability to insert to each candidate slot. i.e. log o(x [e)
({]]))
bog-softmax((a5) ((a))) (2) (......
We apply a binary classification g00/1l-) the last inserted token's representation e, to decide the termination of the generation at each step. During training. only the final step (0/1/en-)) is trained predict I; all intermediate steps trained to predict 0. In the following section, we will continue to discuss how to formulate the final objective function of INSNET using these step-wise distributions.
3.3 Adaptive Parallelization of Insertions
In addition to our efforts for better training efficiency, we also propose an algorithm to adaptively parallelize the generation process of INSNET to speed up the decoding while preserving the generation quality. The idea mimics the graph layerization process the Dinic's algorithm (Dinitz) 2006). Specifically, assume that partition the tokens into different layers, such that the tokens within the

---

Soures: Trebuie ooar sá transfordam cuvintale in tapte.
Source: Trebule dear su transformmrn cirvriteee tegne
ronslating
matter
First
matter
IE
ronslating
layez)
Layerl
lioye22
Jayea
oction
Info
Target: It just a maiter translating words action.
Target: just madler trarmlaiigg words into action
(A) Initial Layerization
(b) Merged Layerization
Figure 5: (a) Initialized layerization of tokens after the pre-training stage. It assumes fully sequential decoding process for now. (b) As the parallelization fine-tuning proceeds, the algorithm gradually merges different layers. Tokens within the same layer are safely parallelizable in decoding without sacrificing log-likelihood over the tolerance threshold T.
same layer safe to insert in parallel without affecting the coherence of context. The algorithm aims to determine the partition such that the log-likelihowe estimation of the resulting parallel insertion- based model is close the sequential insertion-basee models possible. The intuition is that the sequential insertion process maximally captures the inter-dependencies between the output tokens. By staying close to the sequential insertion-based model regarding likelihood prediction ability, we speed up the inference without severely sacrificing the likelihood estimation quality.
We initialize the layerization with sequential insertion order, which is equivalent to single-node layerization, as is illustrated in Figure 5[a) Starting from this initialization, we perform INSNET- Dinic to gradually evolve this fully fully-sequel sequentiau layerization into aa-auurrrrressvve, parallelized layerization, as shown in Figure 5(b) We promote a token I+- from its respective slot (i'r²) in
layer to (i'-1, , (1-1) in the previous layer 1 two conditions are satisfied: 1) In layer 1, slot
(d-11.7 (1))) does not yet have an assigned insertion; 2) log p( log pogr(areee (1-1) 11f 1) S
Here is hyper-paramees that controls how much to parallel. Intuitively, larger usually indicates more tolerance for token likelihood loss caused by parallel decoding. i.e. more parallelization with tradeoff on likelihood estimation accuracy. The choice of is highly mask/data-dependee as it value associated with likelihood. Note that the promotion of tokens only affects the slot representation computation and the insertion likelihood calculation of the respective insertion. When encoding the context (as is discussed in Section 3.TI. we always treat these insertion operations are performed sequentially.
For a target sequence 8cn [Fo 31, m. In-1], after the layerization algorithm, token F4t is layerized in the 1-th layer and be inserted in the slot (i,.)), denote it as w'ig. Suppose have m effective layers total. Denote the corresponding representation for slot (i, given available context in layer I as eif and the transformer's 's encoding of the partial context until the end of layer I as e-1 We denote the token likelihood, position likelihood and termination likelihood of the token #1 }) and g(0/1|e_)) respectively. The general objective of INSNET parameterized by @ can be formulated as:
to be p(=l (e())) ((=r)
L(8,0) =
flog ( + log oo (ot
Σ 1=0
log 2 (1(1= m)
1=0 2
Experiments
We demonstrate the efficiency, flexibility, and model capability of INSNET with two sets of experi- ments. 1) To show INSNET's performance and the flexibility to switch between sequential and parallel decoding on datasets with high inter-token dependency (i.e., less suitable for parallel decoding). We follow the setup in Zhang et al. (2020) and address the unsupervised lexically constrained text generation problem on two datasets Yelp Review and News. 2) To further verify the effectiveness of INSNET-Dinic, we evaluate INSNET (partially) non-autoregressive machine translation model

---

Table 2: Performance comparison Yelp Review and News datasets. For Levenshtein Transformer, insertion and deletion stages are both counted of decoding- iterations. It's non-trivial to do unsupervised lexically constrained text generation with auto-regressive models. To work around this, we implemented a (aon-And-Write(i for better reference. Models with mark re-implemented by us. Other baseline results directly taken from the original papers.
et al. 2019) style auto-regressive transformer-ban model
Yelp Review
News NIST-2/4
Model
BLEU-2/4
NIST-2/4
# Dec. Steps
BLEU-2/4
Dec. Steps
Auto-regressive Transformer
16.68/5.46
2.79/2.86
39.24
8.79/2.40
1.65/1677
36.74
(Plan- (Plan-And- Write static, Yao al. 2019
NMSTG Welleck et
2019
10.06/1.92
1.11/1.12
27.92
10.67/1.58
2.70/2.70
27.85
InDIGO
20190
16.14/4.63
3.08/3.10
45.63
13.89/3.6
3.08/3.10
26.78
(w/ Scarcheu Adaptive Order)
Levenshtein Transformer
14.84/3.96
2.84/2.89
14.28
11.76/1.89
2.64/2.71
16.13
(Parallel Decrdding, (Gu et al. 2019h)
hnST-POINTER-BAE Base (BERT init)
11.48/2.16
2.15/2.15
6.00
12.13/166
2,902.880
6.00
INST-PIINTER Base (BHRT inir+Wiki)
15.63/3.32
3.27/3.30
6.00
13.01/2.51
3.04/3.06
6.00
InST-POINTER-I Large (BERT init+Wiki)
16.78/3.79
3.49/3.53
14.04/3.04
3.28/3.30
6.00
(Ours, Fully-Sequeee Sequential)
19.36/5.78
3.51/3.54
16.31/4.66
3.103.13
INSNET-umiform (Ours)
12.31/2.30
2.19/2.17
7,00
12.89/2.01
2.99/2.90
7,00
INSNET-Dime (Ours, 10.0)
16.73/4.35
3.19/3.20
11.83
14.13/3.75
2.97/3.00
8.13
with parallel decoding three classical datasets: WMT En-En, WMT En-De and WAT En-Ja. The general setup about how hyper-parameters are determined can be found appendix??.
4.1 Lexically Constrained Generation
Experimental Setup Yelp Review dataset consists of 160K training sequences, 10K sequences for validation and 1k test sequences. News dataset consists of 268586 sentences in total, of which 10k are randomly selected as validation set, for testing. YAKE is performed to the test split each dataset to extract lexical constraints.
We vary the hyperparameter the results on both datasets. For position prediction, we are inserting into slots with positions lying in the top 70% of the position distribution mass. For token prediction, are doing top-(1 [1 3. 5}
the range of (10.0, 3.0, 1.0, 0.3, (fuulll-sequential)) and collect
sampling over the vocabulary distribution. The results are shown in Table Figure ((a) and 6(b)
5,78
5.04
4,96
4.71
5.46
4.53
4.59
4.22
0.3
4.63
3.75
4.03
0.3
3.73
3,0
BLEU-
=
10.0
3.96
InsNet POINTER LevT InDIGO
10.0
3,0
3.35
2,51
2,4
3.32
*
2
LevT
Autoregressive
InDIGO
X Autoregressive
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Soquential Ratio
Sequential Ratio
(a) Yelp Review
(b) News
Figure 6: Illustration of the spectral study results terms of BLEEU44/Seeeentil Ratio decoding on Yelp Review and News dataset. Sequential Ratio is computed by Aug. Dec.Steps
Aug. Length
Discussion 1) INSNET is able significantly outperform most previous works the quality-latency trade-off spectrum. With more sequential flavor (i.e. smaller 7), it is generally to achieve the new state-of-the-art in generation quality. Compared to Plan-And- Aan--nn-rrite style autoregressive baseline.
bbttss/ggihubb.om///AAAAAaa

---

Table 3: Generation examples with insertion steps. Every row shows consecutive steps with blue represents the first, second, and third insertion, respectively.
orange
green
Input
day decided
group
Step
the
day decided to started focus on
the local group hurt rule out of
Step
day
decided get started
local group
Step
day, decided to get started focus on the court
the local group burt the government rule out
year
Step
day
get
to focus group
government
the court.
the Just year
Step 15 but on the next day he docided to get started to focus
the local group has beon hurt the government to rule our of
the court For the.
for the rest the last year..
Step 17 but on the next day. he decided to get started to focus on the
the local group has been hurt by the government to rule our of
court for the first time
support for the rest of last year.
INSNET guarantees the perfect incorporation of keywords due to its insertion- based nature. Also, its improved performance is consistent on both datasets. 2) When setting larger to encourage more parallelization of decoding, INSNET is mostly able parallelize the insertions with good preservation of the generative quality. Compared to the vanilla unifornly-paralle setup (INsNET-uniform) INSNET-Dinie significantly improved the effectiveness of parallel decoding.
We also show two samples of the generation process in Table 3 It is inter- esting observe that the model tends to insert punctuation and function words (e.g., Articles and Prepositions) first, and then more concrete content words.
Generation sample
4.2 Non-autoregressive Machine Translation
We train INsNET-Dinic models with r = 2.00 for extended investigation of the performance on machine learning problems. The results in Table 4 show that, INsNET-Dinic is able to achieve comparable or even better performance compared to previous state-of-the-ar non-autoregressive incertionbbseed machine translation models in terms of generation quality and latency.
4.3 Training Time Analysis
To show the training efficiency of the proposed model, we hereby do an empirical and theoretical analysis on the candidate models' training procedure. See Table For empirical results, the statistics collected on the Yelp Review unsupervised lexically constrained text generation problem. All results are collected on a single NVIDIA RTX3090 GPU. The transformer Seq2seq baseline is trained
Table 4: Machine translation evaluation. Each generation iteration of Levenshtein Transformer
requires at least two full executions of the transformer model. Results with a star mark are collected from our re-implementation. Other baseline results are directly taken from original papers. The results for a vanilla transformer is taken from the LevT paper (Gu et al. (019b).
Model
Ro-En
En-De
En-Ja
#Dec Step.
Latency
InDIGO-SAO (wfo KD) Gu el 2019a
32.47
26. 26t14
40.87*
516ms
InDIGO-random (w/ KD) Gu el al. 2019a)
20.20
17.48*
23.91*
502ms
InsT-uniform (+KD) (Stern et 2019
28.52
26.72
41.89*
Olog n) $ 10
107ms
InsT-binary (T = 0.5, +KDJ (temm et al
2019
30,66
27.41
42.17*
Olog n) < 10
92ms
LevT (+KD) et al. 2019b)
33.26
27.27
42.36
(log n) S 2 x 10
116ms
Vanilla Transformer (w/o KD)
32.30
27.17
43.68
389ms
INSNET-uniform
29.13
26.45
41.67
O n) < 10
92ms
INSNET-Dinic (Ours, 2.00)
33.41
27.36
43.71
O(log n) 8 15.8
105ms
KD
33.91
28.05
44.10
O(logn) n)) v 16.1
103ms

---

Table Empirical theoretical comparative study of different algorithms' training efficiency. # of Steps/Seq means during training how many time the transformer needs to be executed per sequence.
Model
Trach//pooc
of Steps/Seq
Auto-regressive Decoder Transformer InDIGO-SAO
35min48s
58min36s 2h12min(SAO)
1 (())
Insertion Transformer-Sequeniie
26h31min
O(n) O(n)
Levenshtein Transformer-Sequentaal
37h24min
Insertion Transformer
8h24min
@ n)
Levenshtein Transformer
16h33min
θ (logg n)
InsT-POINTER
9h36min
e Olog m)
INSNET/INSNET-DE
1h12mis
1
as a Plan-And-Wrice model. POINTER's
training time per epoch is calculated with the official implementation with mixed precision supported by the NVIDIA APEX library 2
Discussion The results in Table 5 confirm the significant improvement training efficiency of INSNET over most previous baselines. In addition, this concretely shows that it is not practically affordable to obtain a sequential insertion- based generator directly with InsT/LevT. Note that SAO of InDIGO is offline, inference-only algorithm that does not require any computation of gradients. Thus its constant factor is significantly smaller than other (nn) According to its original paper and our previous experiments, InDIGO practically needs this process to achieve reasonable performance.
procedures.
4.4 Ablation Study on the Effectiveness of Components
Since the model components of InsNet are closely entangled, it is nontrivial to do ablation study, We design experiments to show that 1). The proposed offset matrix is powerful insertion-oriented position encoding with significantly better capacity. We show that this particularly helps the model learn to terminate the sequence in correct timings. To study this, we truncate the offset matrix to range that it is similar to InDIGO's 's position encoding, and 2) the global
[-1.0, 1.0] (InsNet-trunnatee) representation is necessary for computing the slot representation in INSNET. in general helps to improve the sequence termination performance, and leads to convergence to better local optimum. We report the results of an ablated version of InsNet without global representation (InsNet-noglobal).
The termination NLL (see See 3.3 for its definition) and BLEU scores for each model can be found in appendix Table ??. We also show some random samples from the ablated variants and the failure cases can be find in appendix Table ??.
5 Conclusion & Future Work
We propose INSNET, an efficient and performant insertion- based generator that supports sequential and parallel decoding. Experiments on two unsupervised lexically constrained text generation datasets and three machine translation datasets show the advantages of INSNET over previous methods.
Future work can explore obtaining large-scale pre-trained version of INSNET for further fine-tuning under different downstream scenarios. We anticipate such insertion-based models have better compositional generalizability and controllability.
Acknowledgement
We thank I-Hung Hsu, Dr. Rujun "RJ" Han, Te-Lin Wu, Kai-Wei Chang, Sarik Ghazarian, Alexander Spangher, Yining Hong, Mingyu Derek Ma and all other members from PlusLabNLP/UCLANII group for their participation in initial discussions and comments on paper writing. We would like to thank Huggingface for their great work of the Transformers project. The work is partially supported by Meta SRA and an Amazon research gift.
10

---

References
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning align and translate. 3rd International Conference Learning Representations, ICLR 2015; Conference date: 07-05-2015 Through 09-05-2015.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Lc, and Ruslan Salakhutdinov. 2019. Transformer- XL: Attentive language models beyond a fixed-length context In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.
Yefim Dinitz. 2006. Dinitz' algorithm: The original version and even's version. In Theoretical computer science, pages 218-240. Springer.
Seraphina Goldfarb-Tarramn Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. 2020. Content planning for neural story generation with aristotelian rescoring. In the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 4319-4338.
Jiatao Gu, Qi Liu, and Kyunghyun Cho. 2019a. Insertion- based decoding with automatically inferred generation order. Transactions of the Association for Computational Linguisticss, Lingsisss,,,,,,,,sisssiss,s,, 7:661-676.
Jiatao Gu, Changhan Wang, and Jake Zhao. 2019b. Levenshtein transformer. arXiv preprint
Rujun Han, Hong Chen, Yufei Tian, and Nanyun Peng. 2022. Go back in time: Generating flashbacks in stories with event temporal prompts. In 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
He He, Nanyun Peng, and Percy Liang. 2019. Pun generation with surprise. 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-IILT 2019), volume 1.
Christian F Hempelmann. 2008. Computational humor: Beyond the pun? The Primer of Humor Research. Humor Research, 8:333-360.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017a. Adversarial learning for neural dialogue generation. arXiv preprint arXiv: 1701.06547.
Xiujun Li, Yun-Nung Chen, Lihong Li. Jianfeng Gao, and Asli Celikyilmaz. 2017b. End-to-end task-completion neural dialogue systems. arXiv preprint 17030
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Hisar Manurung, Graeme Ritchie, and Henry Thompson. 2000. Towards a computational model of poetry generation. Technical report, The University of Edinburgh.
Anirudh Mittal, Yufei Tian, and Nanyun Peng. 2022. Ambipun: Generating humorous puns with ambiguous context. In 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), short.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv: 1rXiv:1803.02155.

---

Long-Siang Shih, Wei- Wei-Cheng Chang, and Yiming Yang. 2019. Xl-editor: Post-editing sentences with xinet. arXiv preprint arXiv:1910.104777
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer:
Flexible sequence generation via insertion operations. In International Conference Machine Learning, pages 5976 5985. PMLR.
Jiao Sun, Anjali Narayan-Chen, Shereen Oraby, Shuyang Gao, Tagyoung Chung, Jing Huang, Yang Liu, and Nanyun Peng. 2022. Context-situated pun generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Bowen Tan, Zichao Yang, Maruan Al-Shedivat, Eric P Xing, and Zhiting Hu. 2020. Progressive generation long text. arXiv preprint arXiv:2006. 000222006.1770 15720.
Yufci Tian, Divyanshu Arun Sheth, and Nanyun Peng. 2022. A unified framework for pun generation with humor principles. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Yufei Tian and Nanyun Peng, 2022. Zero-shot sonnet generation with discourse-level planning and aesthetics features. In 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In NIPS.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3156-3164.
Sean Welleck, Kianté Brantley, Hal Daumé lii, and Kyunghyun Cho. 2019. Non-monotonic sequential text generation. In International Conference on Machine Learning, pages 7716-6720 PMLR.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 0048--0577, Lille, France. PMLR.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc Le. 2019, Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.082.
Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan- and-write: Towards better automatic storytelling. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 7378-7385.
Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, and Bill Dolan. 2020.
Pointer: Constrained text generation via insertion-basee generative pre-training arXiv preprint arXiv:2005.00558.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results....
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
12

---

(b) Did you include complete proofs of all theoretical results? [Yes] Most theoretical results are self-explanatory.
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] They will be released upon camera ready.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "vKfn5vUXSTQ", "reviewer": "Reviewer_uPbS", "summary": "The authors propose a new insertion-based model, called INSNET, that supports both sequential and parallel decoding, to tackle the inefficient training of the naive insertion-based model. They experimented on two un-supervised lexically constrained text generation datasets and three machine translation datasets, showing that the INSNET achieves better performance in each task. The proposed approach also greatly reduces the training cost against the previous approach (as reported in Table 4).\n\n\nTypos, or minor revisions for readability:\n\n- at l. 46: \"encodings , i.e.\" -> \"encodings, i.e.\"\n\n- In Table 3, the citation format will be \"Gu et al.\" -> \"(Gu et al.)\" and so on\n\n- Please re-arrange Figures/Tables display in each page. e.g.  Figure 6 could be moved to the later page so that the description will show up on the same page\n\n\n+++++++++\nThank you for addressing my questions! I also checked the uploaded pdf, and I confirm the corresponding descriptions that the authors added accordingly. Yet, the manuscripts will need to be fixed a bit more for the camera-ready, so I will keep my score as \"Weak Accept\".", "questions": "- Data set up for non-autoregressive MT tasks are not found. You need to add them into Appendix accordingly. \n\n- What kind of the error examples did you get in the experiments? Can you report any patterns if exist, or the errors that be specifically caused by the insertion-based approach?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- They conducted experiments on multiple translation tasks, experimentally showing the effectiveness of the proposed approach. In Table 3, they should also report the performance using KD in other method. This seems unfair comparison. Yet, the proposed approach successfully improves the training efficiency.\n\n- Easy to apply the proposed approach to the existing models, by introducing one pass of transformer encoding computation.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VRLLUapjHp", "reviewer": "Reviewer_CKci", "summary": "This article proposed INSNET, an insertion-based text generation model, for efficient training and flexible decoding. Specifically, a new insertion-oriented position encoding method is used in INSNET to enable efficient computation and a hyper-parameter is provided in INSNET to flexibly control the parallel degree. Experiments were performed to demonstrate INSNET’S advantages in terms of training speed, inference efficiency, and generation quality.", "questions": "1. What is the precondition to use this position encoding method, this method seems heavily rely on device support (eg GPU support on matrix operation). Under the precondition, what extra cost would it cause considering it is time-efficient, is it memory efficient compared to other methods?\n\n2. From what Figure 3 illustrated, it seems that the algorithm actually recomputes position encoding in each step, thinking that in each time step, the sequence position is overall recomputed throw matrix operation. In fact, there is no need to recompute previous token position encoding. So, is there any way to go further to avoid this computational waste and maintain efficiency in the same way?\n\n3. t is an important hyper-parameter for INSNET since it can control how much to parallel. It makes sure the flexibility of the model. Experiments should be made to demonstrate that truly influence the parallelism of the model. What is the relationship between and Sequential Ratio?  Are there any formulas that could be given to show how it manipulates the parallel capability of the model?\n\n4. The inference time comparison could also be analyzed to show the model’s efficiency. More analysis about inference time comparison should be given.\n\n5. Compared to the traditional transformer, the training speed is slower, why happen? There should be some reason to be listed in the paper and verified? \n\n6. Since the non-autoregressive transformer is not that common, when introducing INSNET, a figure about it could be given. More explanation could be given to it.\n\n7. Figure 6 shows the performance of INSNET under different Sequential Ratio, could the same experiments be executed for another model to show the overall performance comparison among these models under different Sequential Ratio?\n\n8. The definition of the Loss function contains an error. In the second part of the formula, the symbol l is from 0 to m, while in the first part, l is between 0 and m-1. They are not consistent.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis article proposed a new insertion-based text generation model, which is efficient in time consumption and flexibly supports parallel and sequential decoding.\n\nWeaknesses:\n\nThere lack ablation experiments to verify the effect of three major components of INSNET.\n\nIt is unclear what useful information is captured by one pass of context encoding in INSNET.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "6TX0HO77NV", "reviewer": "Reviewer_EEGj", "summary": "The paper proposes InsNet, an insertainn-based text generation model. It introduces a new insertion-oriented relative position schema for efficient encoding and computation. It introduced an algorithm to adaptively parallelize the generation process to speed up the inference. The paper conducts comprehensive experiments on a few text generation tasks, and the result is competitive. \n", "questions": "What’s KD?\nI can't fully understand Sec 3.2 and 3.3.", "limitations": "Poor writing and presentation. ", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper introduced “offset”, an insertion-oriented relative position encoding schema. It overcomes the problem of prior methods and reduces the re-encoding problem. It works well in practice. The paper also offers an algorithm which decomposes this method into a few fast matrix operations. It’s a novel algorithm which can be widely applied on future insertion based models. \n\nThe paper rigorously and closely compares with prior work. The proposed method works well on most datasets. The results are convincing. \n\nWeakness\nThe paper is very hard to follow. I can’t quite understand Sec 3.2 and 3.3, after spending 1+ hours reading this paper and some of the prior work (InsT and InDIGO). There are many concepts and terminologies in this paper that I can’t fully understand. eg. Dinic’s algorithm, fully-sequential layerization, tau, etc. I even had a hard time to understand the offset algorithm. I am not sure how the model works as a whole system (input, output, modules, training, inference, etc.) I was very excited about the paper, but I am frustrated after spending 2 hours. \nI would give some suggestions:\n1. Use a simpler example. “
 I have a pen . 
” is too long for demonstrating the algorithm. It’s such a pain to read Figure 2, 3, and 4. I believe a sequence like ABCD is visually better and easier to remember.\n2. Elaborate the core equations. There are a good number of equations in Section 3, but it requires decent patience and memory to encode, understand and remember them. Please focus on core/your equations, and illustrate them with figures and proper notation. \n3. Move something to appendix. The paper gets pretty crowded and complicated. You can also consider just get rid of some components. \n4. Show some example outputs. I am curious about how it empirically works and what’s the order of generation. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1b_fGv7Z7y", "reviewer": "Reviewer_iCQn", "summary": "The authors propose an efficient insertion-based text generator (INSNET) that does not require re-encoding of the context. The model follows the insight to enable computation sharing and aggregation. The proposed model is also flexible for the controlled generation and parallel decoding. Experiments on two unsupervised lexically constrained text generation and machine translation tasks verify the model. ", "questions": "Is it possible to conduct an ablation study with only position representation or slot representation? How much impact does each of these elements contribute? ", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n\n1. The proposed model is versatile and intriguing, which contributes to general text generation models. It also achieves a better quality-latency trade-off compared to former edit-based generations. \n2. The novel and well-motivated scheme for position representation and insertion slot representation, which is insightful for the text generation community.\n3. The efficient computation sharing strategy is well-illustrated. The adaptive training scheme is easy to understand as it follows the curriculum learning to start from generic decoding to arbitrary insertion orders. Though it brings certain overhead, experiments show the decoder stays efficient. \n\nweaknesses:\n\n1. Relatively lack of experiments for analysis. \n2. The impact of hyper-parameter $\\tau$ is task-specific and may be costly to tune. The paper should provide some quantitative analysis, e.g., why machine translation can achieve decent quality with smaller $\\tau$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept