SlowGuess/NAACL_2022_OCR huggingface dataset:
Overall, LMTurkers outperform  $S$  with clear margins, evidencing that their annotations can serve as supervisions for training  $S$ . We next conduct iterative training to improve performance of  $S$  on  $\mathcal{T}$  with supervisions from LMTurkers.
5.2 Iterative training
We investigate the effectiveness of LMTurk by simulating scenarios analogous to active learning. Concretely, we compare three schemes of annotating the sampled data  $\mathcal{B}$  at each annotation iteration  $j$ :
- Active learning (AL). We use  $\mathcal{B}$ 's gold labels to show how  $S$  performs with expert annotations. Gold labels are ideal, but costly because expert annotators need to be employed.
- Self training (ST). We use  $S^{j - 1}$ , the model trained in the previous iteration, to annotate  $\mathcal{B}$  (Yarowsky, 1995; Abney, 2004; Lee et al.,
Figure 3: Improving  $S$  with active learning (blue), self training (orange), and LMTurk (green). Free markers at step zero show LMTurker performances; colors distinguish random seeds. Three acquisition functions are: Entropy ( $\bullet$ ), LeastConfident ( $\blacksquare$ ), random sampling ( $\times$ ). At iteration  $j$ , each experiment is repeated three times; we show mean and standard deviation. Appendix Figure 9 visualizes more results.
2013). ST trades supervision quality for annotation cost; no extra cost is introduced. Because there is no external supervision, ST is expected to be a baseline.
- LMTurk. We query the LMTurkers to annotate  $\mathcal{B}$ . LMTurkers are machine learning models, so there is no human labor. Based on the findings in Figure 2, LMTurker supervisions
are expected to have better quality than those of ST. Yet LMTurk could fall behind AL because LMTurker labels are not gold labels.
When sampling  $\mathcal{B}$  from  $\mathcal{U}$  at each iteration  $j$ , we consider the strategies described in §3.3. We employ Random for all three schemes and Entropy/LeastConfident for AL/LMTurk. Entropy and LeastConfident rely on  $S^{j-1}$ . Regarding the number of sampled examples, we experiment with  $|\mathcal{B}| = 100$  and  $|\mathcal{B}| = 400$  for SST2, SST5, AGNews, CoLA. Due to RTE's small size, we use  $|\mathcal{B}| = 20$  and  $|\mathcal{B}| = 100$ . We run for 15 iterations of improving  $\mathcal{S}$ . To aggregate annotations from LMTurkers, we use MajorityVoting (§3.2), which is widely used in crowdsourcing. See §5.3 for a comparison of various aggregation methods.
Figure 3 compares AL, ST, and LMTurk. ST (orange) noticeably helps  $S$  to perform progressively better on AGNews, e.g., when comparing  $S^{15}$  to  $S^0$  shown in the first row, especially when  $|\mathcal{B}| = 400$ . However, we do not identify clear improvements when looking at other tasks. Except for RTE-  $\mathcal{G}^8$ , ST clearly falls behind AL and LMTurk. This inferior performance meets our expectation because there is no external supervision assisting  $S$  to perform better on  $\mathcal{T}$ . In what follows, we omit ST for clearer visualization and discussion.
AL (blue) performs the best in most experiments. However, this comes with extra costs that are not negligible: At each iteration, human annotators need to annotate 100-400 sentences.
LMTurk (green) holds a position between AL and ST on AGNews, SST2, SST5, and CoLA. Somehow surprisingly, LMTurk performs almost comparably to AL on SST2. Unlike AL, LMTurk requires very little human labor; the only human annotation throughout the entire process is the few-shot gold dataset  $\mathcal{G}$ . In contrast, AL has high human annotation cost, e.g., 1000-4000 examples by iteration ten. LMTurk also shows clear performance improvements over ST.
Results on RTE are noisy; we conjecture this is due to its very small test set (277 examples). We do not observe performance improvement of  $S$  along the iterations in experiment RTE- $\mathcal{G}^{32}$ - $|\mathcal{B}| = 100$ , likely due to saturated task performance: TinyBERT-General-4L-312D ( $S$ ) achieves  $66.6\%$  on RTE for the full train set (Jiao et al., 2020).
Comparing sampling strategies. Entropy  $(\bullet)$  and LeastConfident (■) outperform random sampling  $(\times)$  in AGNews and SST2 with noticeable
Figure 4: Comparing strategies of aggregating LMTurker annotations. We compare LMTurk (green) with AL (blue). Strategies: LogitVoting  $(\times)$ , MajorityVoting  $(\blacksquare)$ , WeightedLogitVoting  $(\spadesuit)$ , BestWorker  $(\clubsuit)$ . AL uses gold labels without aggregation  $(\bullet)$ .
margins – for both AL and LMTurk, especially when  $|\mathcal{B}| = 400$ . They also surpass random sampling when using LMTurk for SST5 and CoLA with  $\mathcal{G}^8$ . In other words, Entropy and LeastConfident assist LMTurk to achieve the same performance as of using random sampling, but with fewer annotations. For example in AGNews- $\mathcal{G}^8$ - $|\mathcal{B}| = 100$ , LeastConfident at iteration six already achieves comparable performance as random sampling at iteration eleven. This is economically and environmentally beneficial because the number of queries made to LMTurkers, i.e., the cost of running inference passes on the array of large PLMs, is significantly reduced.
Overall, we show that LMTurk can be used to create datasets for training a specialized model  $S$  of solving  $\mathcal{T}$  in practical scenarios. To reduce computational cost, we use only Entropy in what follows.
5.3 Design choice 1: Aggregation strategies
Figure 4 compares effectiveness of different strategies of aggregating LMTurker annotations (§3.2). Looking at SST5 and AGNews results (top two images), we observe that committee-style aggregation (LogitVoting  $(\times)$ , MajorityVoting  $(\blacksquare)$ , and WeightedLogitVoting  $(\spadesuit)$ ) generally outperforms BestWorker  $(\clubsuit)$ , which simply relies on the LMTurker performing best on  $\mathcal{G}_{dev}$ . LMTurkers perform well on these two datasets as shown by the free markers at iteration zero; ensembling their predictions results in higher-quality datasets.
Figure 5: Running more iterations of improving  $S$  with AL and LMTurk. Sampling strategy Entropy is used for both methods; WeightedLogitVoting is used for aggregating LMTurker annotations.
In contrast, BestWorker  $(\star)$  has stellar performance on RTE (bottom-left), outperforming committee-style aggregation. Note that even the LMTurkers do not perform really well in this experiment, as shown by the free markers at iteration zero - some LMTurkers even perform worse than  $S$ . Ensembling these low-quality annotations seems a worse option than simply relying on the best LMTurker. For CoLA, we observe comparable performance of different aggregation strategies.
5.4 Design choice 2: More iterations
We hypothesize that AL performance is an upper bound for performance when  $S$  is trained with LMTurker annotations – recall that the AL annotations are gold labels. Figure 5 compares AL and LMTurk when running 100 iterations of improving  $S$  on AGNews and 500 iterations on SST2. As expected, AL outperforms LMTurk because the pool of human-annotated data expands. The performance of  $S$  progressively approaches that of the LMTurkers; LMTurk performs comparably to AL in SST2, however, no human labor is required.
5.5 Design choice 3: Distilling logits
We can view LMTurk as a kind of distillation (Hinton et al., 2015): The ability of LMTurkers to solve  $\mathcal{T}$  is progressively transferred to  $S$ . In this section, we explore the utility of distillation: We train  $S$  with predicted logits instead of discrete labels from LMTurkers. Concretely, we train  $S$  by reducing the KL divergence between its predicted probability distribution (over the label set) and the probability distribution from LMTurkers.
Figure 6: Performance of AL and LMTurk with discrete labels  $(\bullet)$  vs. with KL divergence  $(\times)$ . Entropy is used as the sampling strategy and WeightedLogitVoting is used to aggregate worker annotations.
Figure 6 shows that training  $S$  with KL divergence noticeably improves over discrete labels on AGNews and SST5. This is expected: AGNews and SST5 have larger label set size (four and five) such that the probability distribution over the label set is more informative than that of the binary classification tasks SST2 and RTE.
5.6 Design choice 4: Quality-based filtering
One key difference between AL and LMTurk is that LMTurkers are not oracles: Their labels are not perfect. Hence, it is reasonable to consider processing the training data, denoted as  $\mathcal{D}^j$ , for  $S^j$ , instead of using it indiscriminately as in AL.
InstanceTresholding (§3.3.2) preserves annotations in  $\mathcal{D}^j$  for which LMTurkers have the smallest prediction entropy. Concretely, we rank all annotations  $(\mathbf{x},\hat{y},\mathbf{l})\in \mathcal{D}^j$  by entropy(l) and then keep the  $\tau$  percent smallest. Note that we always preserve the human-labeled few-shot data  $\mathcal{G}_{train}$ . We experiment with  $\tau \in \{10\% ,\dots ,90\% ,100\% \}$ .
Figure 7 left shows the performance of  $S$ ; Figure 7 right tracks the status of  $\mathcal{D}^j$ . To measure quality, we compute the accuracy of LMTurker annotations on  $\mathcal{D}^j$  (compared to gold labels); see the lineplots and the left y-axis. We also report the size of  $\mathcal{D}^j$  as scatter plots (right y-axis).
We observe that  $\tau = 10\%$ , i.e., keeping only the  $10\%$  most certain examples, gives the worst performance. This is most obvious at iteration three for SST2: The performance drops to near the majority baseline ( $\approx 50\%$ ). This is because  $\mathcal{D}^3$  is small and
Distilling with intermediate activations likely to further improve performance of  $S$ . However, note that PLM intermediate activations are not always available in a Language-Model-as-a-Service framework.
Figure 7: Training  $S$  with examples for which LMTurkers have low entropy. We report performance of  $S$  (left), number and quality (measured by accuracy) of the preserved examples (right) at each iteration.
unbalanced: It has eight negative (from  $\mathcal{G}^{train}$ ) and 38 positive examples. However, using all the LM-Turker annotations ( $\tau = 100\%$ ) may not be optimal either. This is noticeable when looking at SST5:  $\tau = 90\%$  and  $\tau = 80\%$  are better options.
We see that there is a trade-off between  $\mathcal{D}^j$ 's quality and size from Figure 7 right. Being conservative, i.e., preserving only a handful of annotations from LMTurkers, results in a small, but high-quality  $\mathcal{D}^j$ ; using all the annotations indiscriminately leads to a large  $\mathcal{D}^j$  with low quality. This experiment highlights a key difference between LMTurk and AL: LMTurker annotations are not perfect and taking the annotation quality into consideration when training  $\mathcal{S}$  is crucial.
In this work, our focus is the research question: How to make effective use of current few-shot learners? We propose LMTurk, a simple yet effective method that considers PLM-based few-shot learners as non-expert annotators in crowdsourcing; active learning strategies are incorporated to reduce the cost of annotation. We further show that processing the annotations from LMTurkers can be beneficial.
Future work may combine LMTurker annotations with human annotators in a human-in-the-loop setup (Monarch, 2021) to increase the overall utility of invested resources (Bai et al., 2021). Scaling up to even larger PLMs likely to further boost model performances (Kaplan et al., 2020; Brown et al., 2020) Applying LMTurk to multilingual few-shot learners (Zhao et al., 2021; Winata et al., 2021; Lin et al., 2021) is also promising.
We thank the anonymous reviewers for their insightful comments and suggestions. MZ and HS were supported by the European Research Council (ERC# 740516) and the German Federal Ministry of Education and Research (BMBF, grant #01IS18036A).
Steven Abney. 2004. Understanding the Yarowsky algorithm. Computational Linguistics, 30(3):365-395.
David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Israel Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakatumba-Nabende, Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yimam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Verrah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chukwuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobias Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akinode, Deborah Nabagereka, Maurice Katusiime, Ayodele Awokoya, Mouhamadane MBOUP, Dibora Gebreyohannes, Henok Tilaye, Kelechi Nwaike, Degaga Wolde, Abdoulaye Faye, Blessing Sibanda, Orevaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Salomey Osei. 2021. MasakhaNER: Named Entity Recognition for African Languages. Transactions of the Association for Computational Linguistics, 9:1116-1131.
Omar Alonso, Daniel E. Rose, and Benjamin Stewart. 2008. Crowdsourcing for relevance evaluation. SI-GIR Forum, 42(2):9-15.
Fan Bai, Alan Ritter, and Wei Xu. 2021. Pre-train or annotate? domain adaptation with a constrained budget. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages
5002-5015, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72.
Sangnie Bhardwaj, Samarth Aggarwal, and Mausam Mausam. 2019. CaRB: A crowdsourced benchmark for open IE. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6262-6267, Hong Kong, China. Association for Computational Linguistics.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Chris Callison-Burch. 2009. Fast, cheap, and creative: Evaluating translation quality using Amazon's Mechanical Turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 286-295, Singapore. Association for Computational Linguistics.
Chris Callison-Burch and Mark Dredze, editors. 2010. Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. Association for Computational Linguistics, Los Angeles.
David Cohn, Les Atlas, and Richard Ladner. 1994. Improving generalization with active learning. Machine learning, 15(2):201-221.
David A Cohn, Zoubin Ghahramani, and Michael I Jordan. 1996. Active learning with statistical models. Journal of artificial intelligence research, 4:129-145.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177-190, Berlin, Heidelberg. Springer Berlin Heidelberg.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Anca Dumitrache, Oana Inel, Benjamin Timmermans, Carlos Ortiz, Robert-Jan Sips, Lora Aroyo, and Chris Welty. 2021. Empirical methodology for crowdsourcing ground truth. Semantic Web, 12(3):1-19.
Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020. Active Learning for BERT: An Empirical Study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7949-7962, Online. Association for Computational Linguistics.
Richard M Felder and Rebecca Brent. 2009. Active learning: An introduction. ASQ higher education brief, 2(4):1-5.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, Online. Association for Computational Linguistics.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMLR.
Ben Hachey, Beatrice Alex, and Markus Becker. 2005. Investigating the effects of selective sampling on the annotation task. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 144–151, Ann Arbor, Michigan. Association for Computational Linguistics.
Luheng He, Mike Lewis, and Luke Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 643-653, Lisbon, Portugal. Association for Computational Linguistics.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR.
Jeff Howe. 2008. Crowdsourcing: How the power of the crowd is driving the future of business. Random House.
Jeff Howe et al. 2006. The rise of crowdsourcing. Wired magazine, 14(6):1-4.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan First, and Melvin Johnson. 2020. XTREME: A massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411-4421. PMLR.
Emily Jamison and Iryna Gurevych. 2015. Noise or additional information? leveraging crowdsourced annotation item agreement for natural language tasks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 291-297, Lisbon, Portugal. Association for Computational Linguistics.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, Online. Association for Computational Linguistics.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual LAMA: Investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3250-3258, Online. Association for Computational Linguistics.
Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabagdi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh Rasooli, Sepideh Sadeghi, Erfan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saber Sheybani, Ali Tazarv, and Yadollah Yaghoobzadeh. 2021. ParsiNLU: A suite of language understanding challenges for Persian. Transactions of the Association for Computational Linguistics, 9:1147-1162.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In *ICLR*.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations.
Florian Laws, Christian Scheible, and Hinrich Schütze. 2011. Active learning with Amazon Mechanical Turk. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1546-1556, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Dong-Hyun Lee et al. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896.
Ji-Ung Lee, Jan-Christoph Klie, and Iryna Gurevych. 2021. Annotation curricula to implicitly train non-expert annotators. arXiv preprint arXiv:2106.02382.
Ji-Ung Lee, Christian M. Meyer, and Iryna Gurevych. 2020. Empowering Active Learning to Jointly Optimize System and User Demands. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4233-4247, Online. Association for Computational Linguistics.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
David D Lewis and William A Gale. 1994. A sequential algorithm for training text classifiers. In SIGIR'94, pages 3-12. Springer.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online. Association for Computational Linguistics.