Title: No title found

Abstract: Recent years have seen remarkable progress of text generation in different contexts, including the most common setting of generating text from scratch, the increasingly popular paradigm of retrieval and editing, and others. Text infilling, which fills missing text portions of a sentence or paragraph, is also of numerous use in real life. Previous work has focused on restricted settings, by either assuming single word per missing portion, or limiting to single missing portion to the end of text. This paper studies the general task of text infilling, where the input text can have an arbitrary number of portions to be filled, each of which may require an arbitrary unknown number of tokens. We develop a self-attention model with segment-aware position encoding for precise global context modeling. We further create a variety of supervised data by masking out text in different domains with varying missing ratios and mask strategies. Extensive experiments show the proposed model performs significantly better than other methods, and generates meaningful text patches.

1. INTRODUCTION:

Text generation spans a rich set of tasks that aim to generate natural language from input data. Popular tasks include machine translation, summarization, dialogue, and others. Previous work has made remarkable progress in text generation in various contexts. For example, the most common setting is to generate an entire text sequence from scratch (Mikolov et al., 2010; Sutskever et al., 2014; Bahdanau et al., 2014) . Recent work additionally leverages retrieved reference text to help with generation (Guu et al., 2017; Weston et al., 2018) , and others (Hu et al., 2017; Shen et al., 2017) generate by manipulating specific aspects of given text.

Text infilling, which fills missing text snippets of a sentence or paragraph, is also a common application in real life useful in numerous contexts, such as restoration of historical or damaged documents, contract or article writing with templates, text editing, and so forth. The counterpart application in visual domain is image inpainting (filling missing pixels in images) which has attracted great research and industrial interest and achieved impressive results (Bertalmio et al., 2000; Criminisi et al., 2004; Liu et al., 2018; Yu et al., 2018b) . Text infilling, in contrast, is less explored or has been studied in simplified and more restricted settings. For example, the recent MaskGAN work (Fedus et al., 2018) and the sentence completion task (Zweig & Burges, 2011) have assumed each missing portion of a sentence contains only a single word. The assumption fails to meet the general text infilling need that each part can miss an arbitrary number of tokens and the missing word count is unknown a priori. Other work (Holtzman et al., 2018; Fan et al., 2018) assume the missing text are at the end of a sentence or paragraph, and continuations of the given text are generated. Sun et al. (2017) study image captioning with a single blank surrounded by known text. These studies are not directly applicable to many real scenarios where multiple portions at random positions of the text can be missing.

In this paper, we study the general task of text infilling. Consider input text where an arbitrary number of portions are missing and each portion may originally contain an arbitrary unknown number of tokens. The task aims to infill the missing portions based on the global and surrounding context, to make the text complete and meaningful. For example, given an incomplete sentence (which we call a template) "this is one ____ and i ____ food here", the desired output could be "this is one of my favorite restaurants and i really like the food here". To the best of our knowledge, such general, unconstrained text infilling setting has not been studied previously. We created a variety of supervised data by masking out snippets of sentences with different strategies.

We explore solutions to the task, such as the common attentional sequence-to-sequence model (Bahdanau et al., 2014) and GAN-based approach (Goodfellow et al., 2014) . In particular, to better capture the global and surrounding context of the missing portions, we leverage a multi-head selfattention model (Vaswani et al., 2017) , and devise a segment-aware position encoding mechanism to enable precise localization when there are multiple missing segments and varying number of missing tokens in each.

We conduct extensive experiments in multiple concrete setups, including randomly masked text of varying number of segments and missing ratios. We also study infilling more specific content, such as prepositions and verbs, as well as infilling large portions based on a few anchor words. Results show our proposed model captures semantic meanings and generates patches that fit into the text.

2. RELATED WORK:

Text Generation Neural sequence models such as recurrent network (Mikolov et al., 2010) are commonly used in text generation. Attention mechanism allows the model to search for a set of positions in the source sentence where the most relevant information is concentrated during decoding. Vaswani et al. (2017) propose the Transformer, a model architecture that relies entirely on an attention mechanism (Bahdanau et al., 2014) to draw global dependencies between input and output. New approaches related to deep generative models such as VAE (Kingma & Welling, 2013) and GAN (Goodfellow et al., 2014) have drawn significant attention and give a promising performance in vision domain. When it comes to text generation, Yu et al. (2017) introduce GAN and use policy gradient to train the generative model. Bowman et al. (2015) combine the architecture of a variational autoencoder with the RNN-based language model to explicitly capture global features in a continuous latent variable. Hu et al. (2017) combines VAEs and holistic attribute discriminators for an effective imposition of semantic structures. Generative language models with recursive network structure are usually trained with maximum likelihood in an approach known as teacher forcing. While such approach may reproduce language patterns in training data, its performance is not as promising when sampling on sequences that were never conditioned on during training. To reduce the impact of this problem, Bowman et al. (2015) introduce an RNN-based variational autoencoder generative model that incorporates distributed latent representations of entire sentences. The result is promising when the VAE model is trained with data imputed with missing words and then tested on sentences whose final 20% are imputed. Fedus et al. (2018) introduce an actor-critic conditional GAN that fills in missing text conditioned on the surrounding context to reduce the GAN's mode dropping problem in the textual setting.

Image Inpainting Filling missing pixels in an image, often referred to as image inpainting or image completion, is a task that has attracted much attention (Barnes et al., 2009; Pathak et al., 2016; Iizuka et al., 2017; Yang et al., 2017; Liu et al., 2018; Yan et al., 2018; Yu et al., 2018a) . Current approaches to tackle the hole-filling problem can be divided into two groups. The first set of methods make use of large external databases and attempt to fill in the hole by searching for a patch with similar surroundings (Hays & Efros, 2007) . The other set of methods synthesize the texture with different techniques. Barnes et al. (2009) propose PatchMatch to quickly find approximate nearest neighbor matches between image patches. Pathak et al. (2016) propose Context Encoder to directly predict the missing region with an encoder-decoder CNN. To train this image completion network to be consistent both locally and globally, Iizuka et al. (2017) use global and local context discriminators that are trained to distinguish real images from completed ones. To create a filling patch with finely detailed texture, Yan et al. (2018) introduce a special shift-connection layer to the U-Net architecture while Yang et al. (2017) propose a multi-scale neural patch synthesis approach based on joint optimization of image content and texture constraints.

3. TEXT INFILLING:



3.1. PROBLEM DEFINITION:

In this paper, we consider the problem of text infilling: Given a text template where portions of a body of text are deleted or redacted, we want to fill in the blanks in a way that fit into the global semantic structure and provide meaningful details.

I want some lobster served with cheese , please . I want some __m__ , please .

Original Text ::

Template : 

Data:

In our infilling task setting, a sentence may be viewed as the joint of a set of segments. Each segment consists of multiple tokens in a row. By masking out certain segments(deterministically or stochastically), we receive a template for the input sentence.

Let __m__ be a placeholder for a blank, where multiple tokens in a row are masked out. In Figure 1 , the original text is split into three segments as "I want some", "lobster served with cheese" and ", please .". In the template, the second segment is masked out and replaced by the special token __m__.

Notations:

For better illustration, we introduce the following notations:

For the input sequence x = (s 0 , s 1 , ..., s n ), s i refers to the i th input segment. Let x (i,j) denote the j th token in the i th input segment s i , s i can be represent as (x (i,0) , x (i,1) , ..., x (i,oi) ). The input sequence may also be given as x = (x (0,0) , x (0,1) , ..., x (0,o0) , x (1,0) , x (1,1) , ..., x (1,o1) , ..., x (n,0) , x (n,1) , ..., x (n,on) ).

Let x templatei denote the template sequence that is attended to fill in the blank whose seg_id is i.

We use s i to refer to the filled-in segment for the blank with seg_id = i while x (i,j) denotes a token in it. Finally, let M be the set that contains all the blanks' seg_id.

3.2. APPROACH:

Figure 2 depicts the overall architecture of our model. The basis for our model is a multi-head selfattention token decoder, which fits the task of infilling as it is able to condition on information from both the past and the future. Our implementation replicates Vaswani et al. (2017) .

3.2.1. TEMPLATE:



Initialize Template:

For a discrete input sentence with n segments x = (s 0 , s 1 , ..., s n ), we first generate a template by masking out part of the token segments deterministically or stochastically. The blanks will be replaced by a special token __m__ in the template.

For instance, if we mask out segment i and segment j in x (i < j), then M = {i, j}. The initial template is given as

EQUATION

Update Template:

After filling in each blank, we update the template by replacing the specific placeholder __m__ into corresponding segment. During training, after generating the i th segment, the ground truth s i will be filled back into the template, and x templatei in equation 1 will be updated into x templatej = (s 0 , ..., s i-1 , s i , s i+1 , ..., s j-1 , __m__, s j+1 , ..., s n ). During testing, the inference segment s i will be filled back into the template, and the new template will be x templatej = (s 0 , ..., s i-1 , s i , s i+1 , ..., s j-1 , __m__, s j+1 , ..., s n ). The decoder will attend to the updated template x templatej when filling in next blank, whose seg_id is j.

3.2.2. POSITION ENCODING:

Since the Self-attn architecture based solely on attention mechanism and thus contains no recurrence or convolution, we need to inject additional information about the relative or absolute position of the tokens in the sequence.

As can be seen in Figure 2 , the location of each token in the template can be uniquely determined by its segment number seg_id and the offset in that segment, which we denote as offset_id. As in where i is the dimension and pos = seg_id * base + of f set_id is the unique position index for each token given by (seg_id, of f set_id) and a self-defined integer base.

The positional embeddings have the same dimension d model as the word embeddings, ensuring that the two can be summed. The sum of the positional embeddings and the word embeddings for the input token sequence will be used as input for the Transformer.

3.2.3. APPLICATIONS OF ATTENTION:

As proposed by Vaswani et al. (2017) , an attention function maps a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The input consists of queries and keys of dimension d k , and values of dimension d v . We pack a set of queries, keys and values into matrix Q, K and V representatively to compute the attention function simultaneously. The attention function is given by:

Attention(Q, K, V ) = sof tmax( QK T √ d k )

Multi-head attention mechanism projects queries, keys and value to different representation subspaces and calculates corresponding attention. The attention function outputs are concatenated and projected again before giving the final output. Multi-head attention allows the model to attend to multiple features at different positions.

In this work, the multi-head attention is used in the following two ways: (1) The decoder contains self-attention layers where the keys, values and queries come from the output of the previous layer in the decoder. This allows the decoder to attend to all previous positions and make use of local information during infilling. (2) In "template-decoder attention" layers, the queries come from the previous decoder layer, and the template embeddings are used as memory keys and values. This makes sure the decoder can attend to all positions in the template and capture global semantic information while filling each blank.

3.2.4. TRAINING:

Objective In the infilling process, the decoder will fill in the blanks one by one. For the infilling of each segment, the decoder fills in the missing token auto-regressively, conditioning on the template together with what has been filled in the template. To fill the blank with seg_id = i, the objective is to minimized the following cross-entropy loss:

Li(x (i,0) , x (i,1) , ...,

x (i,o i ) |x template i ) = -log o i j=0 P (x (i,j) |x (i,0) , ..., x (i,j-1) , x template i ), i ∈ M.

The loss L for each infilling sentence is the sum of the cross-entropy loss for each infilling blank:

L = L i , i ∈ M.

Optimizing:

We use Adam optimizer ((Kingma & Ba, 2014)) with β 1 = 0.9, β 2 = 0.997 and = 10 -9 . We follow the setting in Vaswani et al. (2017) and linearly increase the learning_rate for the first warmup_steps training steps, then decrease the learning_rate proportionally to the inverse square root of the step number. We set const = 0.3 and warmup_step = 10000.

learning_rate = const * 1 √ d model * min( step_num ( √ warmup_step) 3 , 1 √ step_num )

4. EXPERIMENTS:

In this section, we evaluate the effectiveness of the multi-head self-attention model for the infilling task in various settings. For all the experiments, we compare our model (Self-attn) to two baselines: attentional sequence-to-sequence model (Seq2Seq) (Bahdanau et al., 2014) and GAN-based approach (GAN) (Goodfellow et al., 2014) . Aside from the recurrent neural network's ability to capture relative position, we inject positional embedding (section 3.2.2) to Seq2Seq and GAN baseline models to keep the settings the same.

4.1. VARYING MASK RATES AND SEGMENTS:

As a start, we conduct a set of tests to find the impact template structure has on infilling performance. We test the performance of different generative models when removing different portions of the tokens. With the same mask rate, we generate templates with a different number of blanks.

Dataset:

We use the Yelp text corpus to train the generative models. This dataset consists of 450k food reviews taken from Yelp. We select positive reviews that contain at most 16 words. The resulting dataset contains 104k sentences for training and 1k sentences for testing with the vocabulary size of 9k. Token segments are masked out randomly in the template.

Quantitative and Human Evaluation:

We use multiple metrics to evaluate the quality of the infilling sentence generated by our model(Self-attn), Seq2Seq and GAN. For quantitative evaluation, we compare the perplexity and the BLEU score Papineni et al. (2002) on the test set.

We also invite ten university students to conduct an unbiased human evaluation. Under each setting (mask rate & mask #segment), each model samples 40 sentences. Samples from different models are randomized in the sample pair. Raters are asked to rank the sentences from the best to the worst in each sample pair. Samples ranking at the 1 st , 2 nd and 3 rd place will receive 3, 2, 1 points respectively.

Table 1 reports the perplexity, BLEU scores and human evaluation scores on Yelp dataset when varying mask rate and the number of mask segments. Our model (Self-attn) outwins Seq2Seq and GAN in all three metrics.

Samples:

Table 2 provides an example of the infilling task. The template, ground truth and three samples generated by Self-attn, Seq2Seq and GAN are listed in the table. While Seq2Seq and GAN generate patches that do not fit with the content (line 3: "at appreciated by chinese food"; line 4: "live right of the app"), Self-attn is able to recover the template in a way that is very similar to the ground truth. Template i live __m__ and i was __m__ chinese food .

4.2. PREPOSITION INFILLING:

Ground Truth i live right down the street and i was craving some good chinese food .

Seq2Seq i live at a ten times and i was at appreciated by chinese food .

GAN i live right of the app and i was looking for chinese food .

Self-attn i live in the neighborhood area and i was impressed with the chinese food . Table 2 : An example from the Yelp data where the template contains two missing portions and 40% of the tokens are masked out.

For this experiment, we use corpus from Grimm's Fairy Tale1 , containing 209 tales collected by the brothers Grimm. We split the long sentences into multiple clauses containing at least 10 but no more than 18 words. The resulting dataset contains 16k sentences for training and 3k sentences for testing with the vocabulary size of 7k.

With the help of the Natural Language Toolkit Bird et al. (2009) , we mask out the prepositions(in, at, on...) and articles(a, an, the) in the corpus. Each template contains three blanks and the average mask_rate is 20.9%. Empty masks that remove nothing will be added to the template if there are less than three segments that satisfy such masking rules.

Quantitative Evaluation:

Since the use-case of prepositions follows a set of rules, and BLEU scores counts matching n-grams in the candidate sentence to n-grams in the reference text, we choose to use BLEU scores to evaluate the generative models' performance for infilling prepositions. Table 3 lists the BLEU score for infilling prepositions on test set. Our model out-performed Seq2Seq and GAN.

Table 4 provides an example of the preposition infilling task. Seq2Seq and GAN are prone to make grammatical mistakes (line 3: "saw at one"; line 4: "the old woman went for, "), which indicates that these two rnn-based generative models failed to grasp the rules of using prepositions. Our model learns the rules and generates prepositions that fit into the template. Ground Truth the old woman went out , but saw no one on the stairs Seq2Seq the old woman went with , but saw at one on the stairs GAN the old woman went for , but saw no one on the stairs Self-attn the old woman went in , but saw that one on the stairs Table 4 : An example from the Grimm Tales data where prepositions are masked out.

4.3. LONGER CONTENT INFILLING:

In this experiment, we test the generative models' performance to fill in longer text contents. We encourage the generative models to generate longer patches by eliminating the information revealed in the template, only a few words are provided as anchors. If all tokens are masked out in the template, then the task reduces to the language model.

In this dataset, we also train the model on Grimm dataset, sentence number and vocabulary size are the same with section 4.2. We only leave one noun and one verb as anchoring words in the template, and the average mask rate is 81.3%.

Perplexity is a commonly-used metric in evaluating the performance of language models. The current task may be regarded as a special language model with a few anchoring words, thus we evaluate the generative models' performance with perplexity. We also use human evaluation to judge the quality of the patches. Human evaluation settings are the same with section 4.1. Table 5 reports the perplexity on train set and test data, as well as human evaluation scores. Our model has the smallest perplexity on both training and testing data.

Two examples of long content infilling are provided in Table 9 . Even though Seq2Seq tends to generate longer patches, its semantic meanings are weak as it only repeats a few words (line 3: "and the little, and the little,"). GAN is flawed with the same shortcoming and failed to generate meaningful patches. In contrast, our model generates a variety of meaningful patches that fit into the global structure.

4.4. MIMIC NEWS GENERATION:

Data-driven news generation for automated journalism (Leppänen et al., 2017 ) is a topic that has attracted much attention. A more challenging task, and also a task with more realistic meanings, is to generate a descriptive sentence from an auxiliary template that contains only a few keywords. We may form such template by keeping the objects or the nouns and removing all the rest.

We choose to use corpus from NBA scripts provided by Wiseman et al. (2017) to simulate news generation. The dataset consists of sentences containing at least 8 but no more than 16 words. The resulting dataset contains 21k sentences for training and 5k sentences for testing with the vocabulary size of 8k. To mimic the task setting of news generation, we provide a player's name or a team's name as well as a set of numbers in the template. Each template contains three blanks and the average mask rate is 78.1%.

The task setting for this experiment is similar to the longer content infilling task in section 4.3. We also use perplexity as the evaluation metric. Table 5 : Perplexity on the train/test sets of Grimm Tales for language models with anchor words.

Template __m__ sound __m__ be __m__

Ground Truth if you bear it without letting a sound escape you , i shall be free

Seq2Seq and sound the be and the little , and the little , and the GAN and sound the be and the , and and

Self-attn the sound said , i will be the king Table 6 : Examples for language models with anchor words.

Two examples for long content infilling are provided in Table 8 . Seq2Seq still suffers from sampling over repeat words (line 3: "defeated the the"). Even though GAN does not generate repeat segments, it fails to learn the script pattern (line 4: "defeated the visiting 114-110"). Our model not only generates meaningful patches, but also provides enough details (line 5: "defeated the Philadelphia_76ers 114-110"), which is suitable for the task of news generation.

Seq2Seq GAN Self-attn Test Perplexity 10.303 7.245 6.538 Train Perplexity 10.675 6.857 6.738

Table 7 : Perplexity on the train/test sets of the NBA scripts for language models with anchor words. Self-attn The Toronto_Raptors defeated the Philadelphia_76ers 114 -110 on Friday .

Template __m__ Bojan __m__ 30 minutes __m__:

Ground Truth Bojan Bogdonavic was not far behind , scoring 22 points in 30 minutes off Seq2Seq Bojan led the way with with points points 30 minutes , while GAN Bojan was second on the team , totaling 19 points , 30 minutes , Self-attn Bojan led the way with 20 points in 30 minutes in the fourth quarter Table 8 : Examples of the NBA scripts for language models with anchor words.

5. CONCLUSION:

We have studied the new general task of text infilling, which aims to fill missing portions of a given sentence or paragraph. The task permits an arbitrary number of missing portions each of which can originally have an arbitrary number of tokens. We develop an efficient self-attention model equipped with a segment-aware position encoding mechanism for precise localization and global context modeling. A variety of supervised datasets are created, with varying mask portions and strategies. Our approach significantly improves over common sequence-to-sequence and GANbased models in the extensive experiments.

:

Grimm's Fairy Tale: https://www.cs.cmu.edu/~spok/grimmtmp/

Figures and Tables:

Figure 1: An example of template creation.

Figure 2: The overall structure of Self-attn. This figure depicts the training process. The decoder will attend to the template at each position, conditioning on the template together with what has been filled in the template. During inference, the input will not go through the masked multi-head attention layer.

Template __m__ Toronto_Raptors __m__ 114 -110 __m__ Ground Truth The Toronto_Raptors defeated the Detroit_Pistons 114 -110 on Sunday at the Air Canada Seq2Seq The Toronto_Raptors defeated the the 114 -110 on Wednesday at the Center GAN The Toronto_Raptors defeated the visiting 114 -110 on Friday .

Quantitative and human evaluations for different mask rates and number of segments.

Table content: <table><tr><td>Mask rate Mask #segments</td><td colspan="3">Metric Template Seq2Seq GAN</td><td>Self-attn</td></tr><tr><td/><td>BLEU Score 63.916</td><td>69.097</td><td>68.470</td><td>71.104</td></tr><tr><td>1</td><td>Perplexity -</td><td>107.480</td><td colspan="2">144.127 38.304</td></tr><tr><td>30%</td><td>Human Eval -</td><td>1.950</td><td>1.775</td><td>2.275</td></tr><tr><td/><td>BLEU Score 42.233</td><td>64.174</td><td>64.337</td><td>65.914</td></tr><tr><td>2</td><td>Perplexity -</td><td>43.044</td><td>36.704</td><td>21.028</td></tr><tr><td/><td>Human Eval -</td><td>1.838</td><td>1.975</td><td>2.188</td></tr><tr><td/><td>BLEU Score 56.838</td><td>61.309</td><td>61.778</td><td>63.543</td></tr><tr><td>1</td><td>Perplexity -</td><td>202.714</td><td colspan="2">230.569 44.864</td></tr><tr><td>40%</td><td>Human Eval -</td><td>2.075</td><td>1.865</td><td>2.055</td></tr><tr><td/><td>BLEU Score 38.279</td><td>55.460</td><td>55.326</td><td>59.192</td></tr><tr><td>2</td><td>Perplexity -</td><td>59.877</td><td>70.195</td><td>25.914</td></tr><tr><td/><td>Human Eval -</td><td>2.005</td><td>1.900</td><td>2.045</td></tr><tr><td/><td>BLEU Score 44.369</td><td>48.865</td><td>48.861</td><td>51.55</td></tr><tr><td>1</td><td>Perplexity -</td><td>244.862</td><td colspan="2">287.415 43.688</td></tr><tr><td>50%</td><td>Human Eval -</td><td>1.725</td><td>1.863</td><td>2.412</td></tr><tr><td/><td>BLEU Score 32.498</td><td>42.613</td><td>42.535</td><td>44.418</td></tr><tr><td>2</td><td>Perplexity -</td><td>99.421</td><td colspan="2">107.558 32.397</td></tr><tr><td/><td>Human Eval -</td><td>1.875</td><td>1.913</td><td>2.238</td></tr></table>

BLEU scores on the Grimm test set for infilling prepositionsTemplate __m__ old woman went __m__ , but saw __m__ one on the stairs

Table content: <table/>

Table 7 lists the perplexity on test set and train set. Our model has the smallest perplexity on both training and testing data.

Table content: <table><tr><td colspan="2">Seq2Seq GAN</td><td>Self-attn</td></tr><tr><td>Test Perplexity 10.411</td><td colspan="2">11.784 9.647</td></tr><tr><td>Train Perplexity 9.768</td><td colspan="2">11.224 8.970</td></tr><tr><td>Human Eval 1.991</td><td>1.338</td><td>2.664</td></tr></table>

Peer review (iclr2019):
 The paper proposes a setting for evaluation of a text infilling task, where a system needs to fill in the blanks in a provided incomplete sentences. The authors select sentences from three different sources, Yahoo Reviews, fairy tales, and NBA scripts, and blank out words with varying strategies, ranging from taking out prepositions and articles to removing all but two anchor words from a sentence. On this data, they compare the performances of a GAN model, Recurrent Seq2seq model with attention, and Transformer model in terms of BLEU, perplexity, and human evaluation. The setting is certainly interesting, and the various data creation strategies are reasonable, but the paper suffers from two main flaws. First, the size of the data set is far from sufficient. Unless the authors are trying to show that the transformer is more data-efficient (which is doubtful), the dataset needs to be much larger than the 1M token it appears to be now. The size of the vocabularies is also far from being representative of any real world setting. More important however is the fact that the authors fail to describe there baseline systems in any details. What are the discriminator and generator used in the GAN? What kind of RNN is used in Seq2seq? What size? Why not use a transformer seq2seq? How exactly is the data fed in / how does the model know which blank it*s generating? It would be absolutely impossible for anyone to reproduce the results presented in the paper. There are some other problems with the presentation, including the fact that contrary to what is suggested in the introduction, the model seems to have access to the ground truth size of the blank (since positional encodings are given), making it all but useless in a real world application setting, but it is really difficult to evaluate the proposed task and the authors* conclusions without a much more detailed description of the experimental setting.