SlowGuess/NAACL_2022_OCR huggingface dataset:
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. *ArXiv* preprint, abs/1910.01108.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multitask prompted training enables zero-shot task generalization.
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021b. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.
Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596-4604. PMLR.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics.
Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2021. UnNatural Language Inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7329-7346, Online. Association for Computational Linguistics.
Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021. Improving and simplifying pattern exploiting training. *ArXiv preprint*, abs/2103.11955.
Shizuo Tsuji and Mary Sutherland. 1980. Japanese Cooking: A Simple Art. Kodansha International.
Prasetya Utama, Nafise Sadat Moosavi, Victor Sanh, and Iryna Gurevych. 2021. Avoiding inference heuristics in few-shot prompt-based finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9063-9074, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275.
Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 217-235, Online. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. ArXiv preprint, abs/2109.01652.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
2 Related Work 2
2.1 Prompt-Based Models 2
2.2 Analyses of Prompts 2
3 Overall Setup 3
4 Effect of Templates 4
4.1 Method 4
4.2 Result 4
4.3 Discussion 6
5 Effect of Target Words 7
5.1 Method 7
5.2 Result 7
5.3 Discussion 8
6 General Discussion 8
6.1 Summary and Interpretation 8
6.2 Alternative Interpretations and Future Directions 9
7 Conclusion 9
A Effect of Punctuation 14
B Details and Lessons from Experiment- ing with GPT-3's API 15
B.1 Choice of Model 15
B.2 Priming vs. Fine-Tuning 15
B.3 Other Tips for Working with GPT-3 16
C Hyperparameters 16
D Compute Used 16
E Additional Figures Discussed in the Main Text 17
F All Prompts 19
F.1 Main Experiment Templates 19
F.2 Ablation Experiment Templates 20
F.3 All Target Words 20
G Aggregated Results 21
G.1 ALBERT on RTE 21
G.2 ALBERT on ANLI R1 22
G.3 T5 770M on RTE 23
G.4 T5 3B on RTE 24
G.5 T0 3B on RTE. 25
G.6 T0 3B on ANLI R1 26
G.7 T5 11B, T0 11B, and GPT-3 175B (Figure 2) 27
H Results of Individual Templates 28
H.1 ALBERT 28
H.2 T0 (3B) 32
H.3 T5 LM-Adapted (3B) 36
I Zero-Shot Results (Figure 4) 40
J Comparison of LM targets, Controlling for the Template 41
K Preliminary Results on HANS 44
L Preliminary Results on Winograd 45
Figure 7: ALBERT on RTE. Note that (1) irrelevant templates slightly outperform the instructive templates, albeit without statistical significance. (2) Irrelevant templates are far worse without quotation and question marks. (3) But there is no significant difference between instructive templates with or without qmarks.
Figure 8: T0 (3B) on RTE. Like ALBERT, irrelevant sans qmarks are significantly worse than irrelevant at each and every shot, but there is no significant difference between instructive with or without qmarks.
A Effect of Punctuation
For irrelevant templates, we find a large effect from the use of quotation and question marks in templates. It is natural to write such punctuation in instructive templates as they help humans to parse an NLI hypothesis as an embedded clause within an instruction sentence (e.g., Given {premise} Should we assume that {"hypothesis}" is true?). For control, we also use quotation and question marks ("qmarks" hereafter) in irrelevant templates where they would not have made sense naturally, e.g., {premise} Single-family zoning is bad for American cities.
Figure 9: T5 LM-Adapted (3B). Unlike the other models, there is no statistical significance between irrelevant with or without qmarks. However, instructive sans qmarks statistically significantly outperform instructive at 32 and 64 shots.
"\{hypothesis\}?" As an ablation, when we remove these qmarks from irrelevant templates, the performance of ALBERT and T0 drops substantially (Figures 7 and 8). In contrast, for T5, qmarks make no difference for irrelevant templates; yet, removing qmarks from instructive templates—where qmarks are natural—boosted performance instead for T5 (Figure 9), but not for T0 nor ALBERT.
Additionally, as a coincidence, most misleading templates contain both quotation and question marks, while most misleading-far templates contain only question marks (Appendix F). But as noted in Section 4.2, there is no consistent pattern between those two misleading categories. In other words, punctuations alone cannot explain everything. As discussed in Section 6.2, the full explanation is likely a combined interactions between the spurious features and the semantics of the templates.
Lastly, note that Schick and Schütze (2021b) and many subsequent papers' prompts for NLI (e.g., "\{hypothesis\}" ? | [mask]. "\{premise\}") are basically null templates with some variation in punctuation between the hypothesis and the premise. We find that models learn poorly with the vanilla \{hypothesis\} [mask] {premise}, but they learn as fast as the instructive templates with Schick & Schütze's punctuated version. That being said, note again that punctuation alone cannot explain the performance gap, since models trained with [mask] \{hypothesis\} {premise} (Fig-
ure 13, pink) perform second to best, yet swapping their premises and hypotheses (Figure 13, purple) makes it the worst performing among all null templates.
B Details and Lessons from Experimenting with GPT-3's API
B.1 Choice of Model
We use the davinci model provided by OpenAI LP's API, which corresponds to the 175 billion parameter model reported in Brown et al. (2020). Concurrent to our work, OpenAI released a new product called the "Instruct Series", but we decided to not experiment with the Instruct Series because no academic paper or technical documentation of any kind is available with the Instruct Series at the time of writing aside from the following claim on their website:
The Instruct models share our base GPT-3 models' ability to understand and generate natural language, but they're better at understanding and following your instructions. You simply tell the model what you want it to do, and it will do its best to fulfill your instructions. This is an important step forward in our goal of building safe models that are aligned with human interests.
Crucially, the Instruct Series is inappropriate for reproducible research because it is unknown what datasets and prompts these models are trained on, and whether any task categories are systematically held out as done by Sanh et al. (2021) and Wei et al. (2021). If it is trained on any prompt or dataset of NLI, it would not be zero-shot, making it an unfair comparison to other models in our experiments. Second, it is still in beta and its training, held-out, and prompt mixtures could change. At least two Instruct Series models were made available in sequence during our writing, and it is not clear if we experiment on an older version, whether it will still be available and reproducible in the future.
B.2 Priming vs. Fine-Tuning
As mentioned in Section 3, we use priming (a.k.a. in-context learning) in lieu of fine-tuning because, at the time of writing, OpenAI's fine-tuning API is limited to 10 runs per month. To train 30 prompts at only two number of shots would take 6 months, assuming we get hyperparameters right at first try. Further, each training run is limited to a maximum of 5 epochs, which often entails an insufficient number steps for few-shot training. We were unable to fine-tune GPT to any reasonable accuracy with our allowed 10 tries in the first month. Finally, the fine-tuning API is limited to GPT variants up to 6.7B, not the 175B model we plan to experiment with.
With priming, we are able to reproduce Brown et al. (2020)'s zero-shot performance on RTE but only with their exact prompt reported in their Figure G.31, all other (even instructive) prompts perform at random at zero shots, suggesting that their reported prompt is highly cherry-picked. We are unable to reproduce their reported few-shot result because they report it at 32 shots, but their API only permits a context length up to 2049 tokens, which is insufficient for RTE. We find that 16 shots are the highest one can reach within the token limit. $^{8}$
Like the gradient updated models, we document the exact examples we use for few-shot priming in our GitHub repository. Unlike the gradient updated models, which are trained on the same  $k$  examples, priming models use different sets of  $k$  priming examples for each inference example (Brown et al., 2020, p. 20). This means that GPT's performance reflects the fact that, overall, it has seen far more than  $k$  examples, making it not directly comparable to the few shots of the gradient updated models. This is not ideal, but our GPT few-shot performance already underperforms what Brown et al. (2020) report, so we choose to not further restrict it to have the same fixed priming examples for all inference examples, which could run into a lack of competence issue (\$6.2) that make its results unusable for our research question.
Lastly, unlike the gradient updated models, we do not run multiple seeds with our GPT experiments because, first, they are expensive. As the API bills by token, using  $k$  shots of priming example effectively multiplies the total cost by  $k$ . Sec-
Depending on the length of the prompt template, 2 or 3 examples still exceed the token limit, in which case we remove one priming example, keeping the other 15 priming examples and the to-be-predicted example unmodified.
6OpenAI never actually discloses which one of their commercially named ada, babbage, curie, davinci "engines" correspond to models of which size. However, Gao et al. (2021a) estimate that they correspond to 350M, 1.3B, 6.7B, and 175B respectively. http://beta.openai.com/docs/engines/instruct-series-beta
ond, OpenAI imposes a monthly quota for each lab, so running multiple seeds will take several more months to complete.
B.3 Other Tips for Working with GPT-3
Using the logprobs argument in their API, we obtain the top 99 predicted target word and their log probabilities. Following Sanh et al. (2021) and Wei et al. (2021), we evaluate by a rank classification of the target words, i.e., if the gold target word is "yes", we consider it as correct as long as the probability of "yes" is higher than that of "no", regardless of whether "yes" is the top-1 prediction generated by the model.
Alarmingly, we find that these top-99 predictions are semantically inconsistent ranked, e.g., for one data example and its top-99 word predictions, it is often the case that, e.g.,  $\mathrm{P}(\mathrm{yes}) > \mathrm{P}(\mathrm{no})$  but  $\mathrm{P}(\mathrm{Yes}) < \mathrm{P}(\mathrm{No})$ . Thus, the choice of the target words' surface form makes a substantial difference in the overall performance. (Not to mention the problem of choosing between yes/no, true/false, correct/incorrect, etc. as studied in Section 5.) OpenAI recommends having no trailing space in the input and let the model predict the first token with a leading space as in "Yes". We find that although stripping the leading space sometimes leads to higher performance for some prompts, overall not applying stripping or other token normalization performs the best.
Another point researchers should pay attention to is the use of what OpenAI calls a "separator" inserted between priming examples. In preliminary experiments, we initially use newline characters as appeared in Brown et al. (2020)'s Appendix G. We later discover that OpenAI recommends using \#\#\ or \n##\#\n as separators. We use the latter and find consistent performance improvement over just using newline characters, and we use it throughout in our main experiments.
For encoder-only models, we follow Schick and Schütze (2021b) and Le Scao and Rush (2021)'s recommendations and use a learning rate of  $1e^{-5}$ . For T5 and T0 models, we follow Raffel et al. (2020) and Sanh et al. (2021)'s recommendations
and use a learning rate of  $1e^{-4}$ . We run several preliminary experiments with learning rates  $(3e^{-4}, 1e^{-4}, 5e^{-5}, 1e^{-5})$  deviating from their recommendations and they perform worse, although our search is not exhaustive due to the high cost of running multiple prompts with multiple random seeds.
Note that T5 and T0 are trained with the Adafactor optimizer (Shazeer and Stern, 2018) in Mesh TensorFlow. Our implementation is in PyTorch, and we find that fine-tuning T5 with PyTorch's implementation of Adafactor yields substantially worse results than the usual choice of the AdamW optimizer. We corresponded with Raffel et al. (2020), who advised us that it might be due to the fact that PyTorch does not have the same learning rate scheduler implementation as TensorFlow's Adafactor does. They recommended us to simply use AdamW, which is what we did. This is somewhat unfortunate because Adafactor is much more memory efficient, which would have drastically reduced the compute resources required and thus enable more comprehensive experiments of the 11B models, which are currently limited to 0 shots and 16 shots only.
Although most models seem to obtain the highest validation accuracy at very early epochs, we train all models to 30 epochs (20 epochs for 11B models) to be safe and select the checkpoint with the highest validation accuracy.
All models use a batch size of 4 with 4 gradient accumulation steps for an effective batch size of 16.
Note that because we use a rank classification of single-token target words, decoding sampling methods (e.g., beam search, top-  $k$ , top-  $p$ ) are unnecessary.
We follow Raffel et al. (2020) and add EOS tokens for input sequences, which yields higher few-shot performance compared to not adding EOS as done by Sanh et al. (2021). However, we omit EOS in the zero-shot setting, which exactly reproduces the results reported by Sanh et al. (2021). See T0's GitHub repository readme<sup>10</sup> for more information.
D Compute Used
Each ALBERT 235M model is trained on a single Nvidia RTX3090. Their main experiments took approximately 192 GPU hours.
$^{10}$ https://github.com/bigscience-workshop/t-zero/tree/master/examples
9 Although sometimes the API returns less than the number of logprobs the user specifies, in which case we contacted OpenAI's customer support who provided us refund by store credit. At the time of publishing, OpenAI now restricts logprobs to a maximum of 5.
Each T5 LMA 770M model is trained on a single A6000. Their main experiments took approximately 48 GPU hours.
The 3B models are each trained by partitioning their layers over four RTX3090s. T5 and T0's main experiments took approximately 2,304 GPU hours in total.
The 11B models are each trained on eight V100s (each with 32GB of memory). T5, T0, and  $\mathrm{T}0 + +$  's main experiments took approximately 1,728 GPU hours in total. (Due to their large GPU memory requirement, we were only able to complete one number of shots.)
E Additional Figures Discussed in the Main Text
Figure 11: ALBERT on RTE. Models trained with irrelevant templates actually slightly outperform the instructive templates, albeit without statistical significance at any number of shots.
Figure 10: How to read these figures: Each dot is the performance of one prompt under one random seed (which controls the sets of few-shot examples) of our baseline model (ALBERT) on RTE validation set. Boxes span from the first quartile to the third quartile, while lines inside boxes mark the medians. Later figures omit the points except outliers in order to improve legibility. See the interactive figures in our GitHub repository or Appendix H for the results of individual prompts.
Figure 12: ALBERT on RTE. There is no statistical significance between misleading-extreme and instructive at any number of shots. In contrast, models trained with misleading-moderate templates are significantly worse than the instructive ones from 16 to 64 shots.
Figure 13: ALBERT on RTE. After 32 shots, models trained with 2 null templates learn just as fast as the instructive templates, but models trained with other null templates (e.g., purple) are much worse.
Figure 14: The best-performing instructive template for T0 (3B) on RTE, {prem} Based on the previous passage, is it true that {"hypo"}? with select LM targets from each category.
Figure 15: Median accuracies of all template-target combinations at 32 shots. In general, the choice of target words (x-axis groups) matters much more than the choice of templates (colors).
Figure 16: T0 (3B)'s 32-shot accuracy with of all template-target combinations on RTE. In general, the choice of target words (x-axis groups) matters much more than the choice of templates (colors).
F All Prompts
F.2 Ablation Experiment Templates
category	template
instructive sans qmarks	{premise} Using only the above description and what you know about the world, {hypothesis}is definitely correct. Yes or no
instructive sans qmarks	{premise} \nquestion: {hypothesis}Yes or no\answer:
instructive sans qmarks	{premise} Are we justified in saying that {hypothesis}
instructive sans qmarks	Given {premise} Should we assume that {hypothesis}is true
instructive sans qmarks	{premise} Based on the previous passage, is it true that {hypothesis}
instructive sans qmarks	Given {premise} Is it guaranteed true that {hypothesis}
instructive sans qmarks	Suppose {premise} Can we infer that {hypothesis}
instructive sans qmarks	Given that {premise} Does it follow that {hypothesis}
instructive sans qmarks	{premise} Question: Does this imply that {hypothesis}
instructive sans qmarks	Given that {premise} Therefore, it must be true that {hypothesis}
irrelevant sans qmarks	{premise} Single-family zoning is bad for American cities. {hypothesis}
irrelevant sans qmarks	{premise} Inflections are annoying and thank god that Middle English got rid of most of them. {hypothesis}
irrelevant sans qmarks	{premise} When Bolyai sent Gauss his discovery of non-Euclidean geometry, Gauss replied that he arrived at the same results 30 years ago. {hypothesis}
irrelevant sans qmarks	{premise} If bonito flakes boil more than a few seconds, the stock becomes too strong. {hypothesis}
irrelevant sans qmarks	{premise} Is the pious loved by the gods because it is pious. Or is it pious because it is loved by the gods. {hypothesis}
Table 4: Used in the study of the effect of question and quotation marks in Appendix A.
F.3 All Target Words
Category	Target Words
yes-no	yes;no
yes-no-like	true;false
yes-no-like	positive;negative
yes-no-like	right;wrong
yes-no-like	correct;incorrect
yes-no-like	agree;disagree
yes-no-like	good;bad
reversed	no;yes
reversed	false;true
reversed	negative;positive
arbitrary	B;C
arbitrary	cat;dog
arbitrary	she;he
Table 5: LM targets used in Section 5. Again, for ternary NLI datasets, we use "unclear" for the neutral class, which performs best after preliminary experiments with other ternary words: "maybe", "sometimes", "perhaps", "possibly", and "neither". Within the arbitrary category, in addition to the common anglophone first names as Le Scao and Rush (2021) use, we also tried word pairs with high semantic similarity ("cat"/"dog"), low similarity ("cake"/"piano", "write"/"sleep"), and pairs which are highly frequent in the English language ("she"/"he", "the"/"a") in preliminary experiments, but we find no consistent difference among these various subcategories of the arbitrary category.
G Aggregated Results
G.1 ALBERT on RTE