SlowGuess/NAACL_2022_OCR huggingface dataset:
Daphne Ippolito $^{1,2}$  Liam Dugan $^{1}$  Emily Reif $^{2}$  Ann Yuan $^{2}$  Andy Coenen $^{2}$  Chris Callison-Burch $^{1}$ $^{1}$ University of Pennsylvania  $^{2}$ Google Research {daphnei, ldugan, ccb} @seas.upenn.edu {ereif, annyuan, andycoenen} @google.co
The task of inserting text into a specified position in a passage, known as fill in the blank (FITB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. While previous work has tackled this problem with models trained specifically to do the fill-in-the-blank task, a more useful model is one that can effectively perform both FITB and continuation. In this work, we evaluate the feasibility of using a single model to do both tasks. We show that models pre-trained with a FITB-style objective are capable of both tasks, while models pre-trained for continuation are not. Finally, we show how FITB models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.
Natural language generation systems are increasingly being incorporated into applications where a human writer and an AI jointly collaborate to construct text. These range from creative domains such as collaborative story writing (Coenen et al., 2021; Akoury et al., 2020) to more practical ones such as email composition and code synthesis (Buschek et al., 2021; Wu, 2018; Austin et al., 2021). These applications are often limited to generating text at the end of what has been written so far. This is because language models (LMs) are typically designed to produce text by repeatedly predicting the next word in a sequence given the previous words. However, there is a need for more powerful interactive tools which enable writers to solicit insertions at any chosen position within the existing text, a task referred to as fill in the blank (FITB) or infilling. For example, a creative writer might want a tool which can insert a description of a place or character, and a programmer might want a system that can fill in the contents of a function located in the middle of their code.
Figure 1: A single model that can handle a variety of related writing tasks is more efficient than separate models per task.
Most prior work tackling FITB consider it a separate task from continuation, one to be specifically optimized for, for example training a model from scratch (Ippolito et al., 2019; Zhu et al., 2019) or finetuning a model trained originally for continuation (Donahue et al., 2020). However, having separate trained models for FITB and for continuation is inefficient for downstream applications where maintaining multiple neural networks can be prohibitive.
Any model that can do FitB can be made to do continuation simply by placing the blank at the end of the input. Thus, in this work we describe how models trained on FitB can be employed effectively for both infilling and continuation operations. We show how T5 (Raffel et al., 2020), one of the most popular pre-trained models, can reasonably handle both tasks, as it was pre-trained with a FitB-like objective. Finetuning T5 further improves its ability and also allows for the incorporation of controllability of generation length and word choice.
2 Supporting FitB and Continuation
Definitions. We define filling in the blank as the task of predicting text to replace a single missing span, usually demarcated with a special token, in an input text passage. (Some prior work considers inputs with multiple blanks, but inserting text at one position at a time better matches the kinds of edits humans do.) We define continuation in the
Findings of the Association for Computational Linguistics: NAACL 2022, pages 2421 - 2432 July 10-15, 2022 ©2022 Association for Computational Linguistics
Example Type	Input	Target
C4FILLBLANK no goal	fill: I love avocados. I ate a sandwich covered in them. _8_I talked to my doctor about it later. It turned out I was allergic to avocados.	After I ate it, my mouth was itchy and tingly.
C4FILLBLANK with goal	fill: I love avocados. I ate a sandwich covered in them. _8_I talked to my doctor about it later. It turned out I was allergic to avocados. Goal: mouth was itchy	After I ate it, my mouth was itchy and tingly.
C4FILLBLANK no goal	fill: I love avocados. I ate a sandwich covered in them. After I ate it, my mouth was itchy and tingly. I talked to my doctor about it later. _8_	It turned out I was allergic to avocados.
C4FILLEND with goal	fill: I love avocados. I ate a sandwich covered in them. After I ate it, my mouth was itchy and tingly. I talked to my doctor about it later. _8_Goal: allergic to	It turned out I was allergic to avocados.
Table 1: Examples of the finetuning objectives. "8" is the approximate length in words of the target sequence. During finetuning, about  $25\%$  of training examples took each of these formats.
traditional language modeling sense-predicting the next token in a sequence given only the previous tokens. Donahue et al. (2020) discuss how language modeling is a special case of infilling, and they use this as justification to finetune a continuation-based language model to do infilling. However, we argue that if continuation is a subtask of infilling, it makes more sense to go in the opposite direction: prioritize a model which can do infilling and check that it achieves satisfactory performance at continuation.
Using a model pre-trained for FitB. T5 is a model pre-trained with a "span corruption" objective very similar to FitB; the model is asked to reconstruct the missing text after random subsequences of the input are replaced with special identifiers. Thus, a pre-trained T5 model can be used without any further training to do both continuation and infilling by appropriately choosing text to mask out. The encoder-decoder architecture of T5 is also more conducive to FitB than the decoder-only architectures that are typically used for continuation-based generation, such as GPT-2 (Radford et al., 2019). This is because the attention mechanism in encoder-decoder architectures allows the context on the left side of the blank to attend to the context on the right, while decoder-only architectures only support masked attention (each token can only attend to the positions to its left).
Even though T5's pre-training objective was a form of FitB, finetuning is still advantageous. For one, our definition of FitB involves only a single masked out substring, not multiple, so finetuning improves alignment with the goal task. Finetuning also allows us to incorporate additional conditioning signals not supported by the pre-trained T5, such as being able to specify the desired length of the generated text or specify words that ought to be included in the blank, a task we refer to as "goal conditioning." Length control, which comes by default in a traditional language model by simply
sampling more or fewer tokens, is particularly necessary for FitB, where the end of the generation must fit seamlessly with the text to its right.
Using a model pre-trained for continuation. The biggest language models available today were largely trained in the continuation rather than the FitB paradigm (Brown et al., 2020b; Black et al., 2021). Since our primary goal is to have a single model for both tasks, we also address the question: if a continuation-trained model is big enough, can it handle FitB without the need for finetuning? Few-shot learning with large language models, as popularized by Brown et al. (2020b), has had success on many tasks in NLP. We try out this approach for FitB by designing a few-shot prompt containing several demonstrations of the FitB task, formulated in a similar "infilling by language modelling" template to that proposed by Donahue et al. (2020). Further details on our approach to selecting a few-shot prompt are in Appendix A.1.
Model. For all experiments with T5, we use the 800M parameter v1.1 'large' model (Appendix A.4 gives additional results from the 3B parameter 'XL' model). To finetune T5 for FitB, we construct training examples from documents by first partitioning the document text into a left context, gap, and right context. The input to the model is then the left and right contexts concatenated with textual representations of the additional conditioning signals. The target sequence is the true text for the blank. This formulation easily supports continuation, as the blank can be deliberately placed at the end (i.e., providing no right context). Finetuning examples are drawn from C4, the same dataset T5 was pre-trained on. Documents are split into word sequences, and these are then randomly truncated to be between 256 and 512 words long. A substring of between 1 and 64 words is selected to be
blanked out. For half of the finetuning examples, the location of the blank is randomly selected, and for the other half, it is always placed at the end. To support length conditioning, we follow Roberts and Raffel (2020) and include a bucketed version of the target length as part of the blank. To support goal conditioning, for half the examples, a random substring of up to half the words of the target is appended to the end of the input. Examples are shown in Table 1.
Baselines We compare T5 against Thoppilan et al. (2022)'s 137B parameter decoder-only language model (referred to in this paper as LLM). Trained explicitly for continuation, this model has been used successfully for few-shot learning in other domains (Austin et al., 2021; Reif et al., 2021). We use the LLM in two ways: (1) as a standard continuation model, prompting with only the left context of an example; and (2) in a few-shot learning paradigm.
Evaluation Datasets We evaluate continuation and FitB on C4 as well as two story writing datasets. We chose this domain because creative writing assistant applications are one of the key areas we expect to benefit from multi-task models (Coenen et al., 2021). Reddit Writing Prompts (RWP) is a corpus of stories from the 'r/WritingPrompts' sub-Reddit (Fan et al., 2018), and we construct validation sets RWPFILLBLANK and RWPFILLEND using the same method described in the previous section. We cap the C4 and RWP validation sets to 5,000 examples each. ROC Stories (ROC) is a crowd-sourced dataset of five-sentence commonsense stories (Mostafazadeh et al., 2016). For ROC Stories, the 2018 validation set is used to construct ROCFILLMIDDLE, where the middle sentence of each story is blanked out, and ROCFILLEND, where the last sentence is blanked out. Unless otherwise noted, all evaluation is done without goal conditioning and uses random sampling with top- $k=50$  as the decoding strategy. Example generations for all evaluation sets can be found at https://bit.ly/2U0Ixxa.
4 Findings
Automatic Evaluation We measure the fluency of proposed generations by evaluating the perplexity of each dataset's examples when the predicted text is placed in the blank (Donahue et al., 2020).
C4FILL BLANK	RWPFILL MIDDLE	ROCFILL BLANK
Few-shot LLM	14.14	19.48	18.21
Pre-trained T5	10.38	14.08	22.62
Finetuned T5	10.33	14.08	20.47
Donahue et al. (2020)	N/A	N/A	23.28
Groundtruth	9.41	12.99	16.90
Table 2: Perplexity of evaluation sets according to LLM when the blank has been filled with approaches involving no fine-tuning (top), finetuned approaches (middle), and the groundtruth (bottom). Lower values indicate that the text was considered more fluent by the LLM.
C4FILL END	RWPFILL END	ROCFILL END
LLM	9.34	12.82	15.55
Pre-trained T5	10.09	13.51	21.71
T5 FILLBLANKCONT	10.04	13.74	19.60
T5 LM-ADAPTION	10.06	13.71	19.68
Groundtruth	9.41	12.99	16.90
Table 3: Perplexity of continuation-based evaluation sets when a continuation has been generated using approaches with no finetuning (top) and two settings of finetuning T5 (middle).
We use the LLM to measure perplexity<sup>1</sup>. The results are shown in Table 2. We see that the LLM struggles to generate fluent infills, even when used in a few-shot setting. The only exception to this is ROC Stories, a dataset with fairly simplistic, predictable language. Finetuning T5 does not result in significantly improved fluency over the pre-trained model except on ROC Stories. Lastly, for ROC Stories, we compare against Donahue et al. (2020)'s finetuned GPT-2 small, which yielded less fluent predictions.
Table 3 shows a similar analysis on our continuation-style datasets. We see that the pretrained T5 generates about as fluent continuations as T5 finetuned in the manner described in Section 3 (T5 FILLBLANKCONT), as well as T5 finetuned for the same number of steps, but only on the continuation task (T5 LM-ADPTION). The first row of Table 3 shows how fluent the LLM scores its own generated continuations.
Human Evaluation Human evaluation was conducted on 70 examples, 35 from RwPFILLBLANK and 35 from RwPFILLEND, with examples about evenly distributed across length buckets. For RwPFILLBLANK evaluation tasks, the rater was presented an input context and several possible sequences that could go in the blank. They were
<sup>1</sup>Note, since this is the same model being used for generation for our continuation baseline, this metric may be biased.
Figure 2: Human ratings of FitB generations (left) and continuation generations (right). Error bars are  $95\%$  confidence intervals.
Finetuned T5	Context	Length
C4FILLBLANK	0.860	0.877
RWPFILLBLANK	0.797	0.881
C4FILLEND	0.858	0.775
RWPFILLEND	0.791	0.746
Table 4: Accuracy of models finetuned on FILL-BLANKCONT at correctly using provided length and goal conditioning signals.
asked to rate each sequence first, on how well it fit the text before it, and second, on how well it fit with the text following it, according to a 5-point slider. For RWPFILLEND, the task was almost the same, except that the rater was presented only a left context and asked to rate how well it continued the prompt. More details are in Appendix A.3. Figure 2 shows the results.
On the FitB task, the pre-trained and finetuned T5 models were indistinguishable in terms of quality. The LLM that formed continuations prompted with only the left context did somewhat better than the few-shot LLM, indicating that few-shot learning is not yet a feasible alternative to finetuning. On the continuation task, the LLM has the highest rating, which is unsurprising since it is a much larger model than T5. However, the finetuned T5 is rated almost as highly. Overall, these results suggest that T5, unlike the LLM, can be used effectively for continuation as well as FitB. Furthermore, if one doesn't care about controllability, pre-trained T5 can be used effectively for both tasks without any further finetuning.
Benefits of Controllability Despite finetuning not resulting in significantly more fluent outputs, there are still good reasons to finetune; namely, finetuning allows for increased controllability. For example, length conditioning is extremely important for FitBmodels, since it is not possible to control
the generation length by simply sampling more or fewer tokens. Pre-trained T5 tends to produce infill proposals which are shorter than the groundtruth (Figure 3), and there is no way to ask the model to produce longer generations. In contrast, finetuned T5 was able to produce generations in the target length bucket over  $74\%$  of the time (Table 4). Goal conditioning, while not strictly necessary for either task, has been shown to be useful for generative commonsense reasoning (Lin et al., 2020) and may empower users in downstream applications such as AI-assisted creative writing (Roemmele, 2021). Finetuned T5 is able to use all of the specified goal words over  $79\%$  of the time.
Domain Transfer Prior work on FitB tends to only evaluate models trained on data from the same domain as the validation set. Our results show that despite training exclusively on C4, T5 models have strong transferability to more targeted domains such as Reddit Writing Prompts. This sort of transferability is extremely important for achieving the goal of having a single model which can handle many tasks and domains.
FITB is a form of Cloze task (Taylor, 1953). Prior deep-learning approaches to this task include training an encoder-decoder model from scratch with length and goal word conditioning (Ippolito et al., 2019); finetuning GPT-2 (Radford et al., 2019; Donahue et al., 2020); and training a custom self-attention architecture on corrupted text (Zhu et al., 2019). None of these show that their fill-in-the-blank models remain effective at continuation or perform well on text domains that differ from the training data. Related to FITB, Mori et al. (2020) investigate a setting where a sentence is randomly deleted from the input, and the model must both predict the location of the deletion as well as its contents. Huang et al. (2020) tackle the sentence infilling task using a mixture of BERT and GPT-2. Lastly, many LM pre-training objectives involve masking out parts of the input then predicting the masked values, which is similar to FITB (Devlin et al., 2019; Raffel et al., 2020; Joshi et al., 2020).
In this work, we make the case for starting with a model capable of filling in the blank when attempting to build a system that can perform both FITB
Figure 3: For each of the FITB validation sets, a histogram of the distribution of sequence lengths (measured in words) of the ground-truth blanked out text and the proposed infills from T5 (after and before finetuning). We see that pre-trained T5 tends to produce text that is shorter than the groundtruth.
and continuation. As LMs become bigger, it will be unsustainable to have separately trained models for each generation task. Multi-task, domain-transferable models, such as the ones we propose, require less total training and are more efficient to store and use at inference time. While pre-trained T5 by itself is capable of both infilling and continuation, additional conditioning signals such as desired length and goal text can be successfully incorporated into fine-tuning in order to support an even greater diversity of model interactions. We focused our experiments on the T5 model; however, we expect that other model families and architectures can be trained similarly to support a variety of generation tasks. For example, GPT-3 (Brown et al., 2020a) recently began supporting "insertion" and "edit" interactions in addition to continuation. Finally, we present a negative result that while few-shot learning is a promising method for building multi-task support without any finetuning, it is challenging to make work for the FitB task.
7 Risks and Limitations
All neural language models, including the ones used in this paper, reflect the biases and other issues present in their training data. Weidinger et al. (2021) discuss these risks in detail. The models and datasets considered in this paper are all in the English, and the proposed methods may work differently in other languages. In addition, the paper mostly focuses on showing results pertinent to the story writing domain; in other domains joint models for continuation and fill-in-the-blank might work worse. Finally, the LLMused in this paper is not publicly available, which to some extent limits reproducibility, though we expect our findings would have been similar had we evaluated with a public model such as GPT-2. We emphasize that
the main contribution of this paper is a comparison of different methods, all of which are easily implementable, rather than new model checkpoints.
Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, and Mohit Iyyer. 2020. Storium: A dataset and evaluation platform for machine-in-the-loop story generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6470-6484.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020a. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020b. Language models are few-shot learners.
Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The impact of multiple parallel phrase suggestions on email input and composition behaviour of native and non-native english writers. In Proceedings
of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-13.
Andy Coenen, Luke Davis, Daphne Ippolito, Ann Yuan, and Emily Reif. 2021. Wordcraft: a humanai collaborative editor for story writing.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Chris Donahue, Mina Lee, and Percy Liang. 2020. Enabling language models to fill in the blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2492-2501.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898.
Yichen Huang, Yizhe Zhang, Ouussama Elachqar, and Yu Cheng. 2020. Inset: Sentence infilling with inter-sentential transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2502-2515.
Daphne Ippolito, David Grangier, Chris Callison-Burch, and Douglas Eck. 2019. Unsupervised hierarchical story infilling. In Proceedings of the First Workshop on Narrative Understanding, pages 37-43.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. Commongen: A constrained text generation challenge for generative commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1823-1840.
Yusuke Mori, Hiroaki Yamane, Yusuke Mukuta, and Tatsuya Harada. 2020. Finding and generating a missing part for story completion. In Proceedings of