SlowGuess/NAACL_2022_OCR huggingface dataset:
the The 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 156-166.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, and Jason Wei. 2021. A recipe for arbitrary text style transfer with large language models. arXiv preprint arXiv:2109.03910.
Adam Roberts and Colin Raffel. 2020. Exploring transfer learning with t5: The text-to-text transfer transformer. Google AI Blog.
Melissa Roemmele. 2021. Inspiration through observation: Demonstrating the influence of automatically generated text on creative writing. pages 52-461.
Wilson L Taylor. 1953. "cloze procedure": A new tool for measuring readability. Journalism Bulletin, 30(4):415-433.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359.
Yonghui Wu. 2018. Smart compose: Using neural networks to help write emails. Google AI Blog.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, pages 12697-12706. PMLR.
Wanrong Zhu, Zhiting Hu, and Eric Xing. 2019. Text infilling. arXiv preprint arXiv:1901.00158.
A.1 Few-Shot Learning Details
Choosing appropriate examples for a few-shot prompt can be challenging as task performance is often sensitive to minor changes in prompt design (Zhao et al., 2021). We experimented with prompts randomly selected from the C4, Reddit Writing Prompts, and ROC Stories training sets, as well as prompts consisting of examples handwritten by the authors with the goal of story-writing in mind. For each prompt source, we randomly generated five possible prompts, each with three examples. To simplify the task, we conditioned on desired length but did not include goal conditioning.
An example prompt is shown in Figure A5. When choosing random few-shot prompts from the dataset train sets, in order to keep the few-shot prompt text within the 512-token context length limit of the LLM(Thoppilan et al., 2022) we used for inference, we only considered examples that contained 100 or fewer tokens, so that the max length of the few-shot prompt was no more than 300 tokens. This left 212 tokens for the text of the actual example we were interested in performing the FitB task on. For our hand-written prompt, we wrote the seven examples shown in Table A8. We generated 5 possible prompts by randomly subsampling 3 examples out of these 7 five times.
Table A6 shows the perplexity of the generations from each few-shot prompt. We note that even leaving room for 212 tokens worth of context text, some evaluation examples did not fit in the prompt length, and these examples were skipped when doing this analysis. Figure A4 shows a histogram of the fraction of validation set examples that remained for each few-shot prompt after the too-long examples were filtered out. Based on these results, we chose to include in human evaluation the best few-shot prompt from ROCFILLMIDDLE and the best few-shot prompt from C4FILLBLANK. Figure 2 in the main paper shows the result from the C4FILLBLANK few-shot prompt, whose outputs were rated slightly higher by human annotators.
Our analysis of few-shot learning prompts was not sufficiently exhaustive to rule out the possibility there might exist a prompt for which this technique would be effective. For example, we did not conduct formal experiments to systematically vary the prompt wording/formatting shown in Figure A5. What we can conclude is that the process of finding an ideal prompt requires time-consuming trial-and-
Figure A4: For many of the (validation set, few-shot prompt) combinations, not all validation set examples fit into the maximum sequence length for the LLM. The x-axis on this figure is the fraction of validation set examples which were retained after too-long examples were filtered out. The y-axis is the count of (validation set, few-shot prompt) pairs.
XL Model	Context	Length
C4FILLBLANK	0.867	0.810
RWPFILLBLANK	0.800	0.830
C4FILLEND	0.864	0.826
RWPFILLEND	0.830	0.820
Large Model	Context	Length
C4FILLBLANK	0.860	0.877
RWPFILLBLANK	0.797	0.881
C4FILLEND	0.858	0.775
RWPFILLEND	0.791	0.746
Table A5: Accuracy of models finetuned on FILL-BLANKCONT at correctly using provided length and goal conditioning signals.
error and is quite difficult!
A.2 Experimenting with Prefix Tuning
During the course of this study, we experimented with the usage of Prefix Tuning (Li and Liang, 2021) for the FitB task. In this method, a fixed-length continuous space prefix is appended to the input sequences and this prefix is directly optimized to maximize performance on a given task. This can be used to estimate an upper bound for the performance of few-shot learning on a given task. We trained two prefixes, both of length 5, on pretrained GPT-2 of size medium (345M) and large (774M) (Radford et al., 2019). While our results showed that the prefix successfully instructed the pre-trained model to perform the FitB task, neither of these models outperformed our few-shot prompts during human evaluation. In fact, they showed only marginally better performance than our random baseline. Due to the discrepancy in
Fill in the blank with about 16 words.  
Text: "We have to leave now!" Sarah shouted. ____ The only way out was up. We climbed flight after flight. The sound of the monsters banging on the door below became more distant but no less threatening.  
Answer: "The zombies are going to break through any moment, and then we'll all be goners."
Fill in the blank with about 32 words. Text: I was minding my business at the park, when I was approached by a little girl who was crying because she had lost _____ so of course I helped search. Answer: her cat, which she had just received for her birthday. She did not want her parents to know she'd already lost him. I'm a good person
Fill in the blank with about 8 words.  
Text: The sun was shining, and little gusts of wind brought through the window ______ shocking contrast from the stale city smells she had grown used to.  
Answer: the faint scents of honeysuckle and freshly turned soil. It was a
Fill in the blank with about 8 words.  
Lina went to see how candy canes were made. She watched as the workers added dye to the hot candy. ____. Finally, they shaped it into a cane and let it cool. Lina felt a new appreciation for candy canes.  
Answer:
Target Continuation
Then, they stretched it out to make it shiny.
Figure A5: In blue, one of the few-shot prompts that was derived from handwritten examples, and in green, the target example we would like to perform infilling on.
parameter count between the prefix tuned GPT-2 models and the LLM model we tested for few-shot prompting, we chose to leave these results out of the final analysis. Future work should seek to explore the limitations of prefix/prompt tuning techniques and the ways in which they and few-shot learning can be fairly compared.
A.3 Finetuning Implementation Details
For length conditioning, when discretizing the target sequence's length to a length bucket, we choose the closest value in  $\{1,2,4,8,16,32,64\}$  to the target's length in words.
All training was done in the Mesh Tensorflow T5 codebase. Each T5 model was finetuned for about 50,000 steps with a batch size of 128 examples (i.e.,  $\sim 6.4\mathrm{M}$  examples were seen during finetuning.) A constant learning rate of 0.0008 was used, and no overfitting was observed.
A.4 Further Finetuning Experiments
In the main paper, we focused on a single finetuning setting, one where half the examples have randomly placed blanks and the other half have blanks always
at the end. We actually experimented with three possible finetuning settings:
- In the standard FILLBLANK setting, the blank location is sampled uniformly randomly across the sequence.
- In the FILLBLANKCONT setting, for half of the examples the blank is randomly selected and for the other half it is always at the end. As we hypothesized that finetuning on such data would result in better performance at the continuation task, this was the setting we used in the main paper.
- In the CONT (a.k.a. LM-ADPTION) setting, the blank is always placed at the end of the sequence. In essence, we are finetuning solely for the continuation objective.
For the FILLBLANKCONT setting from the main paper, we additionally experimented with finetuning a 3B parameter "XL" T5 model.
Table A7 shows the perplexity of all these models on a variety of validation sets. Note that these are perplexities in the conventional definition—perplexity of the target sequence given the input sequence using examples from the validation set—not the fluency measure we report in the main paper.
The perplexity numbers across the different models are comparable, since all models used the default T5 vocabulary. The perplexity numbers across different datasets are not comparable since some datasets, like ROC Stories, are simply easier to model than others. Unsurprisingly, the larger models achieved lower perplexity on all validation sets. We can also see from Table A7 that it was probably not strictly necessary to enforce that  $50\%$  of training examples had blanks at the end. The model finetuned exclusively with randomly placed blanks (FILLBLANK) performed only slightly worse (probably not statistically significant) on the continuation-style validation sets than the FILLBLANKCONT-trained model.
Finally, Table A5 shows the accuracy of both model sizes on the two conditioning signals which were incorporated: length bucket and goal conditioning. Surprisingly, using a larger model improves goal conditioning accuracy but hurts length conditioning accuracy.
A.5 Further Human Evaluation Details
A screenshot of the Human Intelligence Task (HIT) used for annotations is shown in Figure A6. Workers were paid originally paid $1.85 per HIT, but
Few-shot source:	C4FILL BLANK	ROCFILL MIDDLE	RWPFILL BLANK	RWPFILL BLANK-Sent
C4FILLBLANK	15.67	19.72	19.65	16.82
ROCFILLMIDDLE	14.14	19.61	19.48	16.36
RWPFILLBLANK	24.39	20.29	32.33	28.13
RWPFILLBLANK-Sent	18.91	18.21	24.44	19.87
FS CUSTOM	17.98	19.80	21.72	18.38
Finetuned T5 XL	9.99	19.00	13.64	10.03
Finetuned T5 Large	10.33	20.47	14.08	10.37
Table A6: Perplexity of evaluation sets when the blank has been filled in using LLM with few-shot prompting (top) and our best fine-tuned T5 model (bottom). Among the few-shot results, the best method for each dataset is bolded, as well as methods within one standard error.
Pre-trained
model	Finetune setting	C4FILL	ROCFILL	RWPFILL
BLANK	END	MIDDLE	END (T)	BLANK	SENTBLANK
T5 Large	FILLBLANKCONT	11.79	13.47	6.43	6.73	16.15	14.84
T5 Large	FILLBLANK	11.64	13.88	6.41	6.84	16.11	14.89
T5 Large	CONT	16.10	13.26	37.08	6.79	21.35	27.73
T5 XL	FILLBLANKCONT	9.53	11.15	5.34	5.79	13.05	11.98
Table A7: The perplexity according to T5 Large finetuned with three possible training data settings, with blanks placed randomly (FILLBLANK), with blanks placed always at the end (CONT), or with an equal mix of these two (FILLBLANKCONT). For the large-sized models, the one that achieved lowest perplexity on each dataset is bolded.
since the average HIT duration ended up being 15 minutes, we awarded each rater a bonus to raise their pay to an average of $10 per hour. We restricted the HITs to workers for whom Masters had been granted and who had previously done at least 100 HITs.
Each example was shown to three raters, and annotations were rejected if the rater gave a lower overall score to the random output than to the ground-truth one. A total of 3 annotations were rejected. Overall, the Fleiss' kappa agreement of pairs of annotators giving the same numerical score to the same question was 0.26.
Context	Target
An elderly man was sitting alone on a dark path. The man looked down at his feet, and realized ____ . It was a plain pine box and looked as if it had been there for a long time. The man was afraid to look inside the box.	he was holding a bright red box made of pine
The mantle was cluttered with objects: ____ and more than one vase of dried flowers. The bejeweled lamp was at the very back, nearly invisible.	picture frames showing grandchildren and long-ago weddings, knickknacks collected from all over the world,
"We have to leave now!" Sarah shouted. ____ The only way out was up. We climbed flight after flight. The sound of the monsters banging on the door below became more distant but no less threatening.	"The zombies are going to break through any moment, and then we'll all be goners."
The sun was shining, and little gusts of wind brought through the window ____ shocking contrast from the stale city smells she had grown used to.	the faint scents of honeysuckle and freshly turned soil. It was a
I was minding my business at the park, when I was approached by a little girl who was crying because she had lost ____ so of course I helped search.	her cat, which she had just received for her birthday. She did not want her parents to know she'd already lost him. I'm a good person
It was a cold night, and a storm was raging out at sea. A light-ning bolt lit up the sky, briefly illuminating the lighthouse ____ plummeted but just before reaching the churning water, he disappeared in a poof of purple flame!	and the young man peering hesitantly over the sheer cliff. Before the next peal of thunder he jumped. At first he
The magician pulled out of his pocket ____ and then a second one and a third. He didn't stop until soon the ground was covered with them.	a scarlet handkerchief
Table A8: Hand-written fill-in-the-blank examples used for "custom" prompt during few-shot learning.
Instructions for Fill-in-the-blank Evaluation Task
Your goal is to analyze how good an artificial intelligence is at generating text that makes sense with respect to the text before and after it. You will be shown the start of a passage of text where the AI's continuation has been highlighted in yellow. You will then be asked:
Does the highlighted text make sense?
Answer "not at all" if:
- the text is very ungrammatical OR
- though the text is grammatical, the highlighted section makes no sense with respect to the rest of the passage.
Answer "completely" if:
- the text is grammatical and smooth-flowing AND
- the contents of the highlighted section seems completely reasonable given the rest of the passage.
You must adjust every slider to be able to submit the HIT. The questions you've already worked on will be marked with a  $\checkmark$ .
Question 4/8
Jackson checked his tie one last time before entering the room. He wasn't allowed to speak, [...]
Does the highlighted continuation make sense with respect to the text before it?
Now, suppose the text is continued in the following way.
Jackson checked his tie one last time before entering the room. He wasn't allowed to speak, now it was his turn. He glanced around the room, recognizing most of the people instantly. Everyone important was there... this was Jackson's chance. Jackson sat at the computer and opened up his AOL E-mail account, USB keys were strictly forbidden at meetings like this for being unsafe. He downloaded the PowerPoint and opened it.
Does the highlighted section make sense given this continuation?
Figure A6: A screenshot of the question structure for human evaluation.
SUBMIT (This will be enabled once you answer every question.)
Previous Next