Copy That! Editing Sequences by Copying Spans

1 INTRODUCTION . Intelligent systems that assist users in achieving their goals have become a focus of recent research . One class of such systems are intelligent editors that identify and correct errors in documents while they are written . Such systems are usually built on the seq2seq ( Sutskever et al. , 2014 ) framework , in which an input sequence ( the current state of the document ) is first encoded into a vector representation and a decoder then constructs a new sequence from this information . Many applications of the seq2seq framework require the decoder to copy some words in the input . An example is machine translation , in which most words are generated in the target language , but rare elements such as names are copied from the input . This can be implemented in an elegant manner by equipping the decoder with a facility that can “ point ” to words from the input , which are then copied into the output ( Vinyals et al. , 2015 ; Grave et al. , 2017 ; Gulcehre et al. , 2016 ; Merity et al. , 2017 ) . Editing sequences poses a different problem from other seq2seq tasks , as in many cases , most of the input remains unchanged and needs to be reproduced . When using existing decoders , this requires painstaking word-by-word copying of the input . In this paper , we propose to extend a decoder with a facility to copy entire spans of the input to the output in a single step , thus greatly reducing the number of decoder steps required to generate an output . This is illustrated in Figure 1 , where we show how our model inserts two new words into a sentence by copying two spans of ( more than ) twenty tokens each . However , this decoder extension exacerbates a well-known problem in training decoders with a copying facility : a target sequence can be generated in many different ways when an output token can be generated by different means . In our setting , a sequence of tokens can be copied token-bytoken , in pairs of tokens , . . . , or in just a single step . In practice , we are interested in encouraging our decoder to use as few steps as possible , both to speed up decoding at inference time as well as to reduce the potential for making mistakes . To this end , we derive a training objective that marginalises over all different generation sequences yielding the correct output , which implicitly encourages copying longer spans . At inference time , we solve this problem by a variation of beam search that “ merges ” rays in the beam that generate the same output by different means . In summary , this paper ( i ) introduces a new sequence decoder able to copy entire spans ( Sect . 2 ) ; ( ii ) derives a training objective that encourages our new decoder to copy long spans when possible , as well as an adapted beam search method approximating the exact objective ; ( iii ) includes extensive experiments showing that the span-copying decoder improves on editing tasks on natural language and program source code ( Sect . 4 ) . Input . 2 MODEL . The core of our new decoder is a span-copying mechanism that can be viewed as a generalisation of pointer networks used for copying single tokens ( Vinyals et al. , 2015 ; Grave et al. , 2017 ; Gulcehre et al. , 2016 ; Merity et al. , 2017 ) . Concretely , modern sequence decoders treat copying from the input sequence as an alternative to generating a token from the decoder vocabulary , i.e . at each step , the decoder can either generate a token t from its vocabulary or it can copy the i-th token of the input . We view these as potential actions the decoder can perform and denote them by Gen ( t ) and Copy ( i ) . Formally , given an input sequence in = in1 . . . inn , the probability of a target sequence o1 . . . om is commonly factorised into sequentially generating all tokens of the output . p ( o1 . . . om | in ) = ∏ 1≤j≤m p ( oj | in , o1 . . . oj−1 ) ( 1 ) Here , p ( oj | in , o1 . . . oj−1 ) denotes the probability of generating the token oj , which is simply the probability of the Gen ( t ) action in the absence of a copying mechanism.1 When we can additionally copy tokens from the input , this probability is the sum of probabilities of all correct actions . To formalise this , we denote evaluation of an action a into a concrete token as JaK , where JGen ( t ) K = t and JCopy ( i ) K = ini . Using q ( a | o ) to denote the probability of emitting an action a after generating the partial output o , we complete Eq . ( 1 ) by defining p ( oj | o1 . . . oj−1 ) = ∑ a , JaK=oj q ( a | o1 . . . oj−1 ) , i.e . the sum of the probabilities of all correct actions . Modelling Span Copying In this work , we are interested in copying whole subsequences of the input , introducing a sequence copying action Copy ( i : j ) with JCopy ( i : j ) K = ini . . . inj−1 ( indexing follows the Python in [ i : j ] notation here ) . This creates a problem because the number of actions required to generate an output token sequence is not equal to the length of the output sequence anymore ; indeed , there may be many action sequences of different length that can produce the correct output . As an example , consider Fig . 2 , which illustrates all action sequences generating the output a b f d e given the input a b c d e. For example , we can initially generate the token a , or copy it from the input , or copy the first two tokens . If we chose one of the first two actions , we then have the choice of either generating the token b or copying it from the input and then have to generate the token f . Alternatively , if we initially choose to copy the first two tokens , we have to generate the token f next . We can compute the probability of generating the target sequence by traversing the diagram from the right to the left . p ( | a b f d e ) is simply the probability of emitting a stop token and requires no recursion . p ( e | a b f d ) is the sum of the probabilities q ( Gen ( e ) | a b f d ) · p ( | a b f d e ) and q ( Copy ( 4 : 5 ) | a b f d ) · p ( | a b f d e ) , which re-uses the term we already computed . Following this strategy , we can compute the probability of generating the output token sequence by computing 1Note that all occurrences of p ( and q below ) are implicitly ( also ) conditioned on the input sequence in , and so we drop this in the following to improve readability . probabilities of increasing longer suffixes of it ( essentially traversing the diagram in Fig . 2 from right to left ) . Formally , we reformulate Eq . ( 1 ) into a recursive definition that marginalises over all different sequences of actions generating the correct output sequence , following the strategy illustrated in Fig . 2 . For this we define |a| , the length of the output of an action , i.e. , |Gen ( t ) | = 1 and |Copy ( i : j ) | = j − i . Note that we simply assume that actions Copy ( i : j ) with j ≤ i do not exist . p ( ok+1 . . . om | o1 . . . ok ) = ∑ a , ∃ ` .|a|= ` JaK=ok+1 ... ok+ ` q ( a | o1 . . . ok ) · p ( ok+ ` +1 . . . om | o1 . . . ok+ ` ) ( 2 ) Note that here , the probability of generating the correct suffix is only conditioned on the sequence generated so far and not on the concrete actions that yielded it . In practice , we implement this by conditioning our modelling of q at timestep k on a representation hk computed from the partial output sequence o1 . . . ok . In RNNs , this is modelled by feeding the sequence of emitted tokens into the decoder , no matter how the decoder determined to emit these , and thus , one Copy ( i : j ) action may cause the decoder RNN to take multiple timesteps to process the copied token sequence . In causal self-attentional settings , this is simply the default behaviour . We found that using the marginalisation in Eq . ( 2 ) during training is crucial for good results . In initial experiments , we tried an ablation in which we generate a per-token loss based only on the correct actions at each output token , without taking the remainder of the sequence into account ( i.e. , at each point in time , we used a “ multi-hot ” objective in which the loss encourages picking any one of the correct actions ) . In this setting , training yielded a decoder which would most often only copy sequences of length one , as the objective was not penalising the choice of long action sequences explicitly . Our marginalised objective in Eq . ( 2 ) does exactly that , as it explicitly reflects the cost of having to emit more actions than necessary , pushing the model towards copying longer subsequences . Finally , note that for numerical stability purposes our implementation works on the log-probability space as it is common for such methods , implementing the summation of probabilities with the standard log-sum-exp tricks . Modelling Action Choices It remains to explain how we model the per-step action distribution q ( a | o ) . We assume that we have per-token encoder representations r1 . . . rn of all input tokens and a decoder state hk obtained after emitting the prefix o1 . . . ok−1 . This can be the state of an RNN cell after processing the sequence o1 . . . ok ( potentially with attention over the input ) or the representation of a self-attentional model processing that sequence . As in standard sequence decoders , we use an output embedding projection applied to hk to obtain scores sk , v for all tokens in the decoder vocabulary . To compute a score for a Copy ( i : j ) action , we use a linear layer W to project the concatenation ri‖rj of the ( contextualised ) embeddings of the respective input tokens to the same dimension as hk and then compute their inner product : sk , ( i , j ) = ( W · ( ri‖rj ) ) · h > k We then concatenate all sk , v and sk , ( i , j ) and apply a softmax to obtain our action distribution q ( a | o ) . Note that for efficient computation in GPUs , we compute the sk , ( i , j ) for all i and j and mask all entries where j ≤ i. Algorithm 1 Python-like pseudocode of beam search for span-copying decoders . def beam_search ( beam_size ) beam = [ { toks : [ START_OF_SEQ ] , prob : 1 } ] out_length = 1 while unfinished_rays ( beam ) : new_rays = [ ] for ray in beam : if len ( ray.toks ) > out_length or ray.toks [ -1 ] == END_OF_SEQ : new_rays.append ( ray ) else : for ( act , act_prob ) in q ( · | ray.toks ) : new_rays.append ( { toks : ray.toks ‖ JactK , prob : ray.prob * act_prob } ) beam = top_k ( group_by_toks ( new_rays ) , k=beam_size ) out_length += 1 return beam Training Objective We train in the standard supervised sequence decoding setting , feeding to the decoder the correct output sequence independent of its decisions . We train by maximising p ( o | ) unrolled according to Eq . ( 2 ) . One special case to note is that we make a minor but important modification to handle generation of out-of-vocabulary words : iff the correct token can be copied from the input , Gen ( UNK ) is considered to be an incorrect action ; otherwise only Gen ( UNK ) is correct . This is necessary to avoid pathological cases in which there is no action sequence to generate the target sequence correctly . Beam Decoding Our approach to efficiently evaluate Eq . ( 2 ) at training time relies on knowledge of the ground truth sequence and so we need to employ another approach at inference time . We use a variation of standard beam search which handles the fact that action sequences of the same length can lead to sequences of different lengths . For this , we consider a forward version of Eq . ( 2 ) in which we assume to have a set of action sequences A and compute a lower bound on the true probability of a sequence o1 . . . ok by considering all action sequences in A that evaluate to o1 . . . ok : p ( o1 . . . ok ) ≥ ∑ [ a1 ... an ] ∈A Ja1K‖ ... ‖JanK=o1 ... ok ∏ 1≤i≤n q ( ai | Ja1K‖ . . . ‖Jai−1K ) . ( 3 ) If A contains the set of all action sequences generating the output sequence o1 . . . ok , Eq . ( 3 ) is an equality . At inference time , we under-approximate A by generating likely action sequences using beam search . However , we have to explicitly implement the summation of the probabilities of action sequences yielding the same output sequence . This could be achieved by a final post-processing step ( as in Eq . ( 3 ) ) , but we found that it is more effective to “ merge ” rays generating the same sequence during the search . In the example shown in Fig . 2 , this means to sum up the probabilities of ( for example ) the action sequences Gen ( a ) Gen ( b ) and Copy ( 0 : 2 ) , as they generate the same output . To achieve this grouping of action sequences of different lengths , our search procedure is explicitly considering the length of the generated token sequence and “ pauses ” the expansion of action sequences that have generated longer outputs . We show the pseudocode for this procedure in Alg . 1 , where merging of different rays generating the same output is done using group_by_toks .

Summary: This paper study the problem of editing sequences, such as natural language or code source, by copying large spans of the original sequence. A simple baseline solution to this problem is to learn a sequence to sequence neural network, which generates the edited sequence conditioned on the original one. This method can be improved by adding a copying mechanism, based on pointer networks, to copy tokens from the input. However, most of such existing approaches can only copy one input token at a time, which is a limitation when most of the input should be copied, which is the case for most editing tasks. In this paper, the authors propose a mechanism that can copy entire spans of the input instead of just individual tokens. In that case, a particular sequence can often be generated by many different actions (eg. copying individual tokens, pairs of tokens, or the whole span). It is thus important to marginalize over all the actions that generated a particular sequence. This can be done efficiently, using dynamic programming, if the probability of an action depends on the generated tokens only, but not on the sequence of actions used to generate them. In the case of neural network, this means that the decoder of the model takes the tokens as input, instead of the spans. To represent spans, the authors propose to use the concatenation of the hidden states corresponding to the beginning and end of the span. Then the probability of copying a span is obtained by taking the dot product between this representation and the current hidden state of the decoder, and applying the softmax. The authors evaluate the proposed approach on the following tasks: code repair, grammar error correction and edit representations (on wikipedia and c# code).