SlowGuess/ACL_2020_OCR huggingface dataset:
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
R. Rudinger, P. Rastogi, F. Ferraro, and B. V. Durme. 2015. Script induction as language modeling. In Empirical Methods in Natural Language Processing (EMNLP).
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D Manning. 2019. Do massively pretrained language models make better storytellers? arXiv:1909.10705.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi Jaakkola. 2020. Blank language models. arXiv:2002.03079.
Y. Shih, W. Chang, and Y. Yang. 2019. XL-Editor: Post-editing sentences with xlnet. arXiv preprint arXiv:1910.10479.
W. L. Taylor. 1953. "Cloze procedure": A new tool for measuring readability. Journalism Bulletin, 30(4):415-433.
A. Wang and K. Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. arXiv preprint arXiv:1902.04094.
T. Wolf, L. Debut, V. Sanh, J. Chaumont, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew. 2019. HuggingFace's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
L. Yao, N. Peng, R. Weischedel, K. Knight, D. Zhao, and R. Yan. 2019. Plan-and-write: Towards better automatic storytelling. In Association for the Advancement of Artificial Intelligence (AAAI).
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. In NeurIPS.
W. Zhu, Z. Hu, and E. Xing. 2019. Text infilling. arXiv preprint arXiv:1901.00158.
- STORIES (100K examples, 5M words)  
Short stories from the ROCStories dataset (Mostafazadeh et al., 2016). Each story contains a title and five sentences.
- ABSTRACTS (200K examples, 30M words) Abstracts from CS papers on arXiv
- LYRICS (2M examples, 60M words)  
Song lyrics from lyrics.com
We experimented on multiple datasets to demonstrate that our framework was not custom tailored to a single domain. On the STORIES and ABSTRACTS datasets, we include metadata (story title, paper subject matter, etc.), as the first "paragraph" of the document. By providing these paragraphs (Appendix B), our infilling model implicitly learns to summarize (e.g. infill a title given a story), and do conditional generation (e.g. infill a story given a title). On the LYRICS dataset, infilling models may be especially helpful to humans; external aid in the form of rhyming dictionaries is already commonly employed in this domain.
To ensure that all experiments were trained on the same data, we removed infilling examples which would have exceeded our training sequence length of 256 tokens for the model with the longest sequence length (LM-All). This removed no examples from STORIES, a small fraction of examples from LYRICS, and a substantial number of examples from ABSTRACTS.
B Masking function
We design a mask function which takes the entire document and selectively masks several span granularities: words,  $n$ -grams, sentences, paragraphs, and entire documents. Accordingly, models trained via ILM on this masking function offer users the ability to specify the granularity of text to infill at a particular location. This allows users to have coarse but intuitive control over infilling length, so that multiple paragraphs are not generated when the user was expecting a single word.
Our masking function first constructs a tree of the training example (using the natural hierarchy of documents, paragraphs, sentences, and words). Then, using a pre-order tree traversal, each subtree is masked with  $3\%$  probability (or ignored if any of its ancestors are already masked). If the entire document (root node of
the tree) is masked, then the infilling model's job is equivalent to that of a language model. If a word (leaf) is selected to be masked,  $50\%$  of the time we mask that individual word, otherwise we mask an  $n$ -gram of random length between 1 and  $\min(8, \# \text{ words left in the sentence})$  words (inclusive). Note that a word may comprise multiple tokens, as GPT-2 uses sub-word tokenization (Sen-rich et al., 2015). We chose the value of  $3\%$  as, for the datasets we considered, it resulted in a marginal token mask rate of around  $15\%$ , echoing the configuration of Devlin et al. (2019).
We add special tokens for each granularity to our model's vocabulary (e.g. [blank word]), so that the user may specify which granularity they would like the infilling model to produce. This functionality can be explored in our demo: https://chrisdonahue.com/ilm.
While we focus on this specific mask function in this paper, we structured the ILM codebase to allow users to train infilling models for completely different use cases. Users need only define a new mask function which takes complete documents and outputs lists of character-level spans representing the desired spans to be masked.
We use early stopping based on the PPL of the model on infilling the masked token for the validation set. We train all models using the default fine-tuning parameters specified in the transformers library (Wolf et al., 2019), except that we use a batch size of 24 and a sequence length of 256.
Note that the most straightforward way of training an LM on ILM examples (Section 3.2) is to maximize the likelihood of the entire concatenated example:  $\tilde{x}$ , [sep], and  $y$ . This trains the model to predict tokens in  $\tilde{x}$  even though such behavior is not necessary at inference time as  $\tilde{x}$  will always be fully-specified. Nevertheless, we found that this additional supervision improved performance when evaluating model PPL of  $y$ . Conveniently, this is also the default behavior when adapting existing LM training code for use with ILM.
D Evaluation on language modeling and infilling other granularities
Our quantitative evaluation (Section 5) examined the sentence infilling performance of GPT-2 initialized from the large-scale pre-trained checkpoint
STO	ABS	LYR
LM (scratch)	33.4	52.1	25.1
LM-Rev (scratch)	32.9	53.9	24.7
LM-All (scratch)	30.4	44.6	26.2
ILM (scratch)	30.8	45.3	30.6
LM	17.6	25.7	20.8
LM-Rev	25.1	36.7	23.7
LM-All	17.8	25.2	21.5
ILM	18.1	23.9	23.0
Table 3: Document infilling PPL (or language modeling) of ILM and baselines initialized either from scratch or from the pre-trained checkpoint across three datasets. Note that PPL of ILM is similar to LM, implying that our infilling strategy can reasonably maintain the ability to perform language modeling while extending the ability to infill.
STO	ABS	LYR
LM (scratch)	34.0	52.8	28.9
LM-Rev (scratch)	34.9	59.3	30.4
LM-All (scratch)	27.0	46.2	24.3
ILM (scratch)	25.5	46.0	27.5
LM	17.5	25.5	23.9
LM-Rev	26.5	39.0	29.2
LM-All	15.1	24.4	19.3
ILM	14.9	23.5	20.2
after fine-tuning on different datasets and infilling strategies. Here, we report PPL for GPT-2 both initialized from scratch and from the pre-trained checkpoint for several other configurations: language modeling, a mixture of granularities, specific granularities, and language modeling.
D.1 Language modeling
In Table 3, we report PPL for "document infilling," which is equivalent to language modeling (because  $\tilde{x}$  is always [blank document]). Because of how we structured our mask function (Appendix B),  $3\%$  of infilling examples consist of the entire document masked out, which results in the ability of our ILM framework to perform standard infilling. We see that performance of ILM is similar to that of LM on this task, even though ILM sees far fewer examples of language modeling compared to LM.
Table 4: Mixture infilling PPL of all models (a mixture of all granularities).
STO	ABS	LYR
LM (scratch)	35.6	51.5	25.1
LM-Rev (scratch)	34.8	65.1	24.7
LM-All (scratch)	33.4	45.0	26.2
ILM (scratch)	34.3	45.3	30.6
LM	18.3	24.2	20.8
LM-Rev	26.5	42.8	23.7
LM-All	20.4	23.4	21.5
ILM	20.7	22.5	23.0
Table 5: Paragraph infilling PPL of all models.
STO	ABS	LYR
LM (scratch)	36.0	65.4	33.5
LM-Rev (scratch)	35.1	92.2	35.8
LM-All (scratch)	27.1	53.8	27.1
ILM (scratch)	26.7	51.0	31.0
LM	18.3	27.9	27.7
LM-Rev	27.1	46.5	34.3
LM-All	15.6	22.3	21.4
ILM	15.6	22.4	22.6
Table 6: Sentence infilling PPL of all models.
D.2 Mixture of granularities
In Table 4, we report results for a mixture of granularities. Specifically, we run the same mask function we use for training (Appendix B) on our test data and evaluate PPL on the masked spans. This reflects general infilling ability across a wide variety of granularities (and hence lengths). Unlike our other quantitative evaluations, there may be multiple variable-length spans missing from each example in this evaluation. Results are similar to that of sentence infilling. Namely, that ILM outperforms LM and LM-Rev and is similar to LM-All despite using much less memory.
D.3 Individual granularities
In Tables 5 to 8 we report PPL values for infilling performance on paragraphs, sentences, n-grams, and words, respectively, across the three datasets.
For each granularity, we create one infilling example per document from the test set with exactly one masked span (randomly chosen from all spans of that granularity for that document). Then, we compute PPL only on the tokens which comprise the masked span, i.e., PPL is computed for all models on exactly the same set of tokens. Across all granularities, we observe that ILM outperforms
STO	ABS	LYR
LM (scratch)	36.1	62.5	34.1
LM-Rev (scratch)	36.4	89.1	36.3
LM-All (scratch)	26.4	60.1	24.3
ILM (scratch)	23.1	49.5	26.3
LM	19.2	25.5	28.2
LM-Rev	26.6	45.0	34.8
LM-All	14.5	20.5	18.6
ILM	13.8	21.5	18.8
Table 7: N-gram infilling PPL of all models.
STO	ABS	LYR
LM (scratch)	32.3	57.2	34.8
LM-Rev (scratch)	31.6	100.0	36.7
LM-All (scratch)	12.6	51.8	12.5
ILM (scratch)	9.2	37.9	12.2
LM	17.1	23.0	28.7
LM-Rev	24.1	45.0	35.1
LM-All	7.5	15.8	9.5
ILM	5.4	14.2	8.5
LM and LM-Rev and either outperforms or is comparable with LM-All while using less memory.
E Details on human evaluation
For human evaluation, we sampled 100 stories from the test set of the STORIES dataset. From each story, we masked out one sentence at a time, thereby resulting in 500 stories with masked sentences. Then we used these stories as context and tasked each model with infilling the masked sentence.
We compared 8 models in total. In addition to the four models reported in Section 6 (BERT, SA, LM, and ILM), we included the models which are initialized from scratch (as opposed to initialized from the large-scale pre-trained checkpoint) for exhaustive comparison. Furthermore, to filter out spam, we used a control model which always generates "This sentence was generated by a computer." Lastly, we included the original sentence from the dataset as a reference model (Human) to sanity check the max score is around  $80\%$ .
Each annotator was shown 8 stories, one from each model, and was asked to identify one of the five sentences generated by machine (see Figure 4 for an example). Among the 100 collected responses, we filtered out 5 responses whose annota
tion for the control model was wrong. The quantitative and qualitative results can be found in Table 9 and Figure 5, respectively. All model outputs and responses of human evaluation can be found at https://github.com/chrisdonahue/ilm.
Table 8: Word infilling PPL of all models.
Score (%)
Control	0
BERT	20
SA	29
LM (scratch)	40
LM	41
ILM (scratch)	39
ILM	45
Human	78
Table 9: Human evaluation results.
Identify one of the five sentences generated by machine.
Patty was excited about having her friends over.
She had been working hard preparing the food.
Patty knew her friends wanted pizza.
All of her friends arrived and were seated at the table.
Patty had a great time with her friends.
Figure 4: Example of a task and instruction for human evaluation on Amazon Mechanical Turk.
Lily always loved to read.  
She wondered sometimes,  
what it would be like to write a book?  
[blank]  
Lily did well in the course, and during it,  
wrote a short book.
BERT I held her hand and helped her sit.
SA Of her, but she didn't know her.
LM She practiced reading a lot every week.
ILM Finally, in middle school, her teacher introduced her to writing that.
Human She decided to take a course on fiction writing.
Yesterday was Kelly's first concert. She was nervous to get on stage. [blank]
Kelly was then happy.  
She couldn't wait to do it again.
BERT Or rather, what the next job would be now.
SA I was going out I was going to the beach.
LM I put on about thirty sugar cubes.
ILM The issues are getting so many people crazy. Human I could never catch up and each week got worse.