Text Infilling

1 INTRODUCTION . Text generation spans a rich set of tasks that aim to generate natural language from input data . Popular tasks include machine translation , summarization , dialogue , and others . Previous work has made remarkable progress in text generation in various contexts . For example , the most common setting is to generate an entire text sequence from scratch ( Mikolov et al. , 2010 ; Sutskever et al. , 2014 ; Bahdanau et al. , 2014 ) . Recent work additionally leverages retrieved reference text to help with generation ( Guu et al. , 2017 ; Weston et al. , 2018 ) , and others ( Hu et al. , 2017 ; Shen et al. , 2017 ) generate by manipulating specific aspects of given text . Text infilling , which fills missing text snippets of a sentence or paragraph , is also a common application in real life useful in numerous contexts , such as restoration of historical or damaged documents , contract or article writing with templates , text editing , and so forth . The counterpart application in visual domain is image inpainting ( filling missing pixels in images ) which has attracted great research and industrial interest and achieved impressive results ( Bertalmio et al. , 2000 ; Criminisi et al. , 2004 ; Liu et al. , 2018 ; Yu et al. , 2018b ) . Text infilling , in contrast , is less explored or has been studied in simplified and more restricted settings . For example , the recent MaskGAN work ( Fedus et al. , 2018 ) and the sentence completion task ( Zweig & Burges , 2011 ) have assumed each missing portion of a sentence contains only a single word . The assumption fails to meet the general text infilling need that each part can miss an arbitrary number of tokens and the missing word count is unknown a priori . Other work ( Holtzman et al. , 2018 ; Fan et al. , 2018 ) assume the missing text are at the end of a sentence or paragraph , and continuations of the given text are generated . Sun et al . ( 2017 ) study image captioning with a single blank surrounded by known text . These studies are not directly applicable to many real scenarios where multiple portions at random positions of the text can be missing . In this paper , we study the general task of text infilling . Consider input text where an arbitrary number of portions are missing and each portion may originally contain an arbitrary unknown number of tokens . The task aims to infill the missing portions based on the global and surrounding context , to make the text complete and meaningful . For example , given an incomplete sentence ( which we call a template ) “ this is one ____ and i ____ food here ” , the desired output could be “ this is one of my favorite restaurants and i really like the food here ” . To the best of our knowledge , such general , unconstrained text infilling setting has not been studied previously . We created a variety of supervised data by masking out snippets of sentences with different strategies . We explore solutions to the task , such as the common attentional sequence-to-sequence model ( Bahdanau et al. , 2014 ) and GAN-based approach ( Goodfellow et al. , 2014 ) . In particular , to better capture the global and surrounding context of the missing portions , we leverage a multi-head selfattention model ( Vaswani et al. , 2017 ) , and devise a segment-aware position encoding mechanism to enable precise localization when there are multiple missing segments and varying number of missing tokens in each . We conduct extensive experiments in multiple concrete setups , including randomly masked text of varying number of segments and missing ratios . We also study infilling more specific content , such as prepositions and verbs , as well as infilling large portions based on a few anchor words . Results show our proposed model captures semantic meanings and generates patches that fit into the text . 2 RELATED WORK . Text Generation Neural sequence models such as recurrent network ( Mikolov et al. , 2010 ) are commonly used in text generation . Attention mechanism allows the model to search for a set of positions in the source sentence where the most relevant information is concentrated during decoding . Vaswani et al . ( 2017 ) propose the Transformer , a model architecture that relies entirely on an attention mechanism ( Bahdanau et al. , 2014 ) to draw global dependencies between input and output . New approaches related to deep generative models such as VAE ( Kingma & Welling , 2013 ) and GAN ( Goodfellow et al. , 2014 ) have drawn significant attention and give a promising performance in vision domain . When it comes to text generation , Yu et al . ( 2017 ) introduce GAN and use policy gradient to train the generative model . Bowman et al . ( 2015 ) combine the architecture of a variational autoencoder with the RNN-based language model to explicitly capture global features in a continuous latent variable . Hu et al . ( 2017 ) combines VAEs and holistic attribute discriminators for an effective imposition of semantic structures . Generative language models with recursive network structure are usually trained with maximum likelihood in an approach known as teacher forcing . While such approach may reproduce language patterns in training data , its performance is not as promising when sampling on sequences that were never conditioned on during training . To reduce the impact of this problem , Bowman et al . ( 2015 ) introduce an RNN-based variational autoencoder generative model that incorporates distributed latent representations of entire sentences . The result is promising when the VAE model is trained with data imputed with missing words and then tested on sentences whose final 20 % are imputed . Fedus et al . ( 2018 ) introduce an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context to reduce the GAN ’ s mode dropping problem in the textual setting . Image Inpainting Filling missing pixels in an image , often referred to as image inpainting or image completion , is a task that has attracted much attention ( Barnes et al. , 2009 ; Pathak et al. , 2016 ; Iizuka et al. , 2017 ; Yang et al. , 2017 ; Liu et al. , 2018 ; Yan et al. , 2018 ; Yu et al. , 2018a ) . Current approaches to tackle the hole-filling problem can be divided into two groups . The first set of methods make use of large external databases and attempt to fill in the hole by searching for a patch with similar surroundings ( Hays & Efros , 2007 ) . The other set of methods synthesize the texture with different techniques . Barnes et al . ( 2009 ) propose PatchMatch to quickly find approximate nearest neighbor matches between image patches . Pathak et al . ( 2016 ) propose Context Encoder to directly predict the missing region with an encoder-decoder CNN . To train this image completion network to be consistent both locally and globally , Iizuka et al . ( 2017 ) use global and local context discriminators that are trained to distinguish real images from completed ones . To create a filling patch with finely detailed texture , Yan et al . ( 2018 ) introduce a special shift-connection layer to the U-Net architecture while Yang et al . ( 2017 ) propose a multi-scale neural patch synthesis approach based on joint optimization of image content and texture constraints . 3 TEXT INFILLING . 3.1 PROBLEM DEFINITION . In this paper , we consider the problem of text infilling : Given a text template where portions of a body of text are deleted or redacted , we want to fill in the blanks in a way that fit into the global semantic structure and provide meaningful details . Data In our infilling task setting , a sentence may be viewed as the joint of a set of segments . Each segment consists of multiple tokens in a row . By masking out certain segments ( deterministically or stochastically ) , we receive a template for the input sentence . Let __m__ be a placeholder for a blank , where multiple tokens in a row are masked out . In Figure 1 , the original text is split into three segments as `` I want some '' , `` lobster served with cheese '' and `` , please . '' . In the template , the second segment is masked out and replaced by the special token __m__ . Notations For better illustration , we introduce the following notations : For the input sequence x = ( s0 , s1 , ... , sn ) , si refers to the ith input segment . Let x ( i , j ) denote the jth token in the ith input segment si , si can be represent as ( x ( i,0 ) , x ( i,1 ) , ... , x ( i , oi ) ) . The input sequence may also be given as x = ( x ( 0,0 ) , x ( 0,1 ) , ... , x ( 0 , o0 ) , x ( 1,0 ) , x ( 1,1 ) , ... , x ( 1 , o1 ) , ... , x ( n,0 ) , x ( n,1 ) , ... , x ( n , on ) ) . Let xtemplatei denote the template sequence that is attended to fill in the blank whose seg_id is i . We use s′i to refer to the filled-in segment for the blank with seg_id = i while x ′ ( i , j ) denotes a token in it . Finally , let M be the set that contains all the blanks ’ seg_id . 3.2 APPROACH . Figure 2 depicts the overall architecture of our model . The basis for our model is a multi-head selfattention token decoder , which fits the task of infilling as it is able to condition on information from both the past and the future . Our implementation replicates Vaswani et al . ( 2017 ) . 3.2.1 TEMPLATE . Initialize Template For a discrete input sentence with n segments x = ( s0 , s1 , ... , sn ) , we first generate a template by masking out part of the token segments deterministically or stochastically . The blanks will be replaced by a special token __m__ in the template . For instance , if we mask out segment i and segment j in x ( i < j ) , then M = { i , j } . The initial template is given as xtemplate = xtemplatei = ( s0 , ... , si−1 , __m__ , si+1 , ... , sj−1 , __m__ , sj+1 , ... , sn ) ( 1 ) Update Template After filling in each blank , we update the template by replacing the specific placeholder __m__ into corresponding segment . During training , after generating the ith segment , the ground truth si will be filled back into the template , and xtemplatei in equation 1 will be updated into xtemplatej = ( s0 , ... , si−1 , si , si+1 , ... , sj−1 , __m__ , sj+1 , ... , sn ) . During testing , the inference segment s′i will be filled back into the template , and the new template will be xtemplatej = ( s0 , ... , si−1 , s ′ i , si+1 , ... , sj−1 , __m__ , sj+1 , ... , sn ) . The decoder will attend to the updated template xtemplatej when filling in next blank , whose seg_id is j .

Summary: The paper proposes a setting for evaluation of a text infilling task, where a system needs to fill in the blanks in a provided incomplete sentences. The authors select sentences from three different sources, Yahoo Reviews, fairy tales, and NBA scripts, and blank out words with varying strategies, ranging from taking out prepositions and articles to removing all but two anchor words from a sentence. On this data, they compare the performances of a GAN model, Recurrent Seq2seq model with attention, and Transformer model in terms of BLEU, perplexity, and human evaluation.