SlowGuess/NAACL_2021_OCR huggingface dataset:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6035-6044, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. Computing Research Repository, arXiv:1907.11692.
Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Quanlu Zhang, Yaming Yang, Yunhai Tong, and Jing Bai. 2020. LadaBERT: Lightweight adaptation of BERT through hybrid model compression. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3225-3234, Barcelona, Spain (Online). International Committee on Computational Linguistics.
David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152-159, New York City, USA. Association for Computational Linguistics.
Juri Opitz. 2019. Argumentative relation classification as plausibility ranking. In Preliminary proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers, pages 193-202, Erlangen, Germany. German Society for Computational Linguistics & Language Technology.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Association for Computational Linguistics.
Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. Computing Research Repository, arXiv:1912.10165.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, NeurIPS 2019.
Victor Sanh, Thomas Wolf, and Alexander Rush. 2020. Movement pruning: Adaptive sparsity by finetuning. In Advances in Neural Information Processing Systems, volume 33, pages 20378-20389. Curran Associates, Inc.
Timo Schick and Hinrich Schütze. 2020. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze questions for few shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, Kyiv, Ukraine (Online). International Committee on Computational Linguistics.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2020a. Green AI. Commun. ACM, 63(12):54-63.
Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. 2020b. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6640-6651, Online. Association for Computational Linguistics.
H Scudder. 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363-371.
Pierre Stock, Angela Fan, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. 2021. Training with quantization noise for extreme model compression. In International Conference on Learning Representations.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics - on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743-758.
Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. Computing Research Repository, arXiv:1806.02847.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30-36, Minneapolis, Minnesota. Association for Computational Linguistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Jason Weston and Chris Watkins. 1999. Support vector machines for multi-class pattern recognition. In *ESANN*, volume 99, pages 219-224.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Ji Xin, Rodrigo Nogueira, Yaoliang Yu, and Jimmy Lin. 2020. Early exiting BERT for efficient document ranking. In Proceedings of SustainNLP: Workshop on Simple and Efficient Natural Language Processing, pages 83-88, Online. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 5753-5763. Curran Associates, Inc.
David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, pages 189-196, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8BERT: Quantized 8bit BERT. In NeurIPS EMC2 Workshop.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. Computing Research Repository, arXiv:1810.12885.
Our implementation can be found in the supplementary material. It extends the original implementation of PET by Schick and Schütze (2021) which, in turn, is based on the Transformers library (Wolf et al., 2020) and PyTorch (Paszke et al., 2017). All dependencies are listed in requirements.txt. Detailed instructions on how our results can be reproduced using this implementation can be found in README.md.
Unless explicitly stated differently, we use the exact same set of hyperparameters as Schick and Schütze (2021) (Table 7) with the only difference that for iPET, we only train 3 generations of models to speed up training. All of our experiments were conducted using a single GPU with 11GB RAM (NVIDIA GeForce GTX 1080 Ti). With this GPU, training a single PET model for 250 steps took approximately 45 minutes. Depending on the task, labeling unlabeled examples took 0.2-1.5 hours per model. Training the final classifier for 5,000 steps on the soft-labeled dataset took 2.5 hours on average. Below, we list task-specific implementation details for all tasks in SuperGLUE.
COPA For COPA, we randomly switch the two options  $c_{1}$  and  $c_{2}$  during training with a probability of  $50\%$  to make the input more diverse; for inference, we always keep the original order. For distilling the final PET model, we obtain logits for unlabeled examples  $x$  from individual PVPs  $\mathbf{p}$  as
$s_{\mathbf{p}}(y \mid x) = \log q_{\mathbf{p}}(y \mid x)$ ; we use the input format proposed by Liu et al. (2019).
WiC Similar to COPA, we randomly switch the input sentences  $s_1$  and  $s_2$  during training. Given a word  $w$  and two sentences  $s_1$  and  $s_2$ , we use the sequence  $w: s_1 \mid s_2$  as input for the final sequence classification model, where | marks the boundary between two text segments.
WSC Unlike other SuperGLUE tasks, the WSC formulation of Raffel et al. (2020) and Brown et al. (2020) requires free-form completion, meaning that for each sentence  $s$  and pronoun  $p$ , we only have a single correct choice  $n$  that the model needs to predict, but we do not provide any alternatives. During training, we thus use regular cross entropy loss between  $n$  and  $\tilde{q}_{\mathbf{p}}(n \mid s, p)$  as defined in Eq. 4. However, in many cases this would allow the LM to easily identify the correct target based on the number of masks provided, so we modify each target by randomly adding up to three additional mask tokens, for which we require the model to predict a special <pad> token. For inference, we always just add a single mask token to ensure consistent results across multiple evaluations and perform greedy decoding as described in Section 3. We then follow Raffel et al. (2020) to map the output produced by the LM to a label  $y \in \{\text{true}, \text{false}\}$ . For distillation, given an unlabeled example  $x$  we set  $s_{\mathbf{p}}(y \mid x) = 1$  if the model's output for  $x$  was mapped to  $y$  and  $s_{\mathbf{p}}(y \mid x) = 0$  otherwise. We provide inputs to the final PET model in the format  $s \mid n$  where  $|$  is the boundary between two text segments and mark  $p$  in  $s$  with asterisks.
MultiRC Deviating from the hyperparameters used by Schick and Schütze (2021), we use a maximum sequence length of 512 tokens for MultiRC both during training and inference because we found many passages to be much longer than 256 tokens. Input for the final sequence classification model is of the form  $p \mid q \mid a$  where  $p$  is the passage,  $q$  is the question,  $a$  is the answer candidate and we use | to mark boundaries between text segments.
ReCoRD For ReCoRD, we again use a maximum sequence length of 512 because many passages require more than 256 tokens. For some questions  $q$ , the ReCoRD training set contains a huge number of answer candidates. To facilitate training, we split each example into multiple examples as follows: let  $C$  be the set of answer candidates
Parameter	Value
adam_epsilon	1e-8
gradient Accumulation_steps	8
learning_rate	1e-5
max_grad_norm	1.0
max_seq_length	256
pet_max_steps	250
sc_max_steps	5,000
per_gpu_train_batch_size	2
distillation_temperature	2
weight Decay	0.01
Table 7: Hyperparameters for PET from Schick and Schütze (2021)
Dataset	Metrics	|Unlabeled|	|Dev|	|Test|
BoolQ	Acc.	9,427	3,270	3,245
CB	Acc./F1	20,000	57	250
COPA	Acc.	400	100	500
MultiRC	F1a/EM	5,100	953	1,800
ReCoRD	F1/EM	20,000	10,000	10,000
RTE	Acc.	20,000	278	300
WiC	Acc.	6,000	638	1,400
WSC	Acc.	554	104	146
Table 8: Important statistics for all datasets used
with  $C^+ \subset C$  being the set of correct answers. We create a training example for each  $c \in C^+$  by randomly selecting up to 9 negative examples from  $C \setminus C^+$  for a total of 10 answer candidates.
For each task and number of examples  $t$ , we create the FewGLUE training set  $\mathcal{T}$  by shuffling the entire original training set with a fixed random seed and collecting the first 32 examples of the shuffled dataset. Following (Raffel et al., 2020; Brown et al., 2020), we select only positive examples for WSC; for both MultiRC and ReCoRD, we follow Brown et al. (2020) and select a total of 32 questions – which corresponds to more than 32 training examples – to enable a fair comparison with GPT-3.
The unlabeled datasets for all tasks are obtained by collecting up to 20,000 examples from their training sets and removing the labels. As the training sets for RTE and CB are very small, for both tasks we additionally select random unlabeled examples from the MNLI training set for a total of 20,000 examples. For evaluation, we use the official validation and test sets for all tasks that are available at https://super.gluebenchmark.com/tasks. All datasets included in SuperGLUE are in English. Additional details for each dataset are given in Table 8.
Preprocessing We do not perform any preprocessing, except shortening all examples to the maximum sequence length. This is done using the longest first strategy implemented in the Transformers library. All input sequences are truncated before applying patterns.