Strength in Numbers: Estimating Confidence of Large Language Models by Prompt Agreement
Year: 2023

Abstract:
Large language models have achieved impressive few-shot performance on a wide variety of tasks. However, in many settings, users require confidence estimates for model predictions. While traditional classifiers produce scores for each label, language models instead produce scores for the generation which may not be well calibrated. We compare generations across diverse prompts and show that these can be used to create confidence scores. By utilizing more prompts we can get more precise confidence estimates and use response diversity as a proxy for confidence. We evaluate this approach across ten multiple-choice questionanswering datasets using three models: T0, FLAN-T5, and GPT-3. In addition to analyzing multiple human written prompts, we automatically generate more prompts using a language model in order to produce finer-grained confidence estimates. Our method produces more calibrated confidence estimates compared to the log probability of the answer to a single prompt. These improvements could benefit users who rely on prediction confidence for integration into a larger system or in decisionmaking processes. https://github.com/JHU-CLSP/ Confidence-Estimation-TrustNLP2023

heading: Future Work
text: We leave this to future work. Limitations
The main limitation of this work is the lack of human evaluation. Since confidence scores are typically used for model explainability, a practical evaluation of scores from our method would be a human-in-the-loop scenario where a user is tasked with understanding a system and making decisions based on the output. The primary questions for this human study would be to determine if our scores are more useful to users than other methods, such as log probabilities, and would being presented with confidence scores lead to different decisions. Second, we focused on multiple-choice questions, with a specific set of possible options. Since MC QA and classification are so similar, our analysis of many MC QA datasets is sufficient to show that our method works for text classification. However, there are other use cases for these models that do not have pre-determined answer choices, such as open-ended questions or summarization. Third, while we supported our decision to only use the largest models in each model family due to their superior performance, we acknowledge that replicating our study across different model sizes (e. g. , FLAN-T5-Small, -Base, -Large, -XL, -XXL), is useful for ensuring our method is robust to the number of parameters. Further, we acknowledge a drawback of our method is the difficulties in comparing across other calibration techniques since the Rand index scores are not normalized. While there are ways to normalize the scores (see Section 7), we decided against these methods in our evaluation because they were either against our zero-shot setting or heavily influenced ECE results based on how scores were normalized. We leave these questions to future work. References Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel
Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. 2022. Ask Me Anything: A simple strategy for prompting language models. ArXiv:2210. 02441 [cs]. Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Alshaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike
Tian-jian Jiang, and Alexander Rush. 2022. PromptSource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 93–104, Dublin, Ireland. Association for Computational Linguistics. Philip Bachman, Ouais Alsharif, and Doina Precup. 2014. Learning with pseudo-ensembles. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005. 14165 [cs]. ArXiv: 2005. 14165. Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. 2022. On the relation between sensitivity and accuracy in in-context learning. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210. 11416. Shrey Desai and Greg Durrett. 2020. Calibration of Pre-trained Transformers. arXiv:2003. 07892 [cs]. ArXiv: 2003. 07892. Jorge Díez Peláez, Juan José del Coz Velasco, Antonio Bahamonde Rionda, et al. 2006. A support vector method for ranking minimizing the number of swapped pairs. Li Dong, Chris Quirk, and Mirella Lapata. 2018. Confidence modeling for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 743–753, Melbourne, Australia. Association for Computational Linguistics. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816–3830, Online. Association for Computational Linguistics. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321–1330. PMLR. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. arXiv preprint arXiv:1909. 00277. Abhyuday Jagannatha and hong yu. 2020. Calibrating Structured Output Predictors for Natural Language Processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2078–2092, Online. Association for Computational Linguistics. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering. Transactions of the Association for Computational Linguistics, 9:962–977. Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133–142. Alexander Johansen and Richard Socher. 2017. Learning when to skim and when to read. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 257–264, Vancouver, Canada. Association for Computational Linguistics. Amita Kamath, Robin Jia, and Percy Liang. 2020. Selective Question Answering under Domain Shift. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5684– 5696, Online. Association for Computational Linguistics. Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082–8090. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. 2020. Calibrated Language Model Fine-Tuning for In- and Out-ofDistribution Data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1326–1340, Online. Association for Computational Linguistics. Aviral Kumar and Sunita Sarawagi. 2019. Calibration of encoder decoder models for neural machine translation. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. Khanh Nguyen and Brendan O’Connor. 2015. Posterior calibration and exploratory analysis for natural language processing models. Alexandru Niculescu-Mizil and Rich Caruana. 2005. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning, ICML ’05, pages 625– 632, New York, NY, USA. Association for Computing Machinery. Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. 2019. Measuring calibration in deep learning. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906. 02361. William M Rand. 1971. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846–850. Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. 2020. Getting closer to ai complete question answering: A set of prerequisite real tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8722–8731. Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. 2016. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multitask Prompted Training Enables Zero-Shot Task Generalization. arXiv:2110. 08207 [cs]. ArXiv: 2110. 08207. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904. 09728. Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics. Aili Shen, Daniel Beck, Bahar Salehi, Jianzhong Qi, and Timothy Baldwin. 2019. Modelling uncertainty in collaborative document quality assessment. In Proceedings of the 5th Workshop on Noisy Usergenerated Text (W-NUT 2019), pages 191–201, Hong Kong, China. Association for Computational Linguistics. Taylor Shin, Yasaman Razeghi, Robert L. Logan, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. Taylor Sorensen, Joshua Robinson, Christopher Rytting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022. An information-theoretic approach to prompt engineering without ground truth labels. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. Dream: A challenge data set and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217–231. Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019a. Quarel: A dataset and models for answering questions about qualitative relationships. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7063– 7071. Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019b. Quartz: An open-domain dataset of qualitative relationship questions. arXiv preprint arXiv:1909. 03553. Niket Tandon, Bhavana Dalvi Mishra, Keisuke Sakaguchi, Antoine Bosselut, and Peter Clark. 2019. Wiqa: A dataset for" what if. " reasoning over procedural text. arXiv preprint arXiv:1909. 04739. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211. 09085. Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xiaodong He, and Bowen Zhou. 2019. Multi-hop reading comprehension across multiple documents by reasoning over heterogeneous graphs. arXiv preprint arXiv:1905. 07374. Lijing Wang, Dipanjan Ghosh, Maria Teresa Gonzalez Diaz, Ahmed Farahat, Mahbubul Alam, Chetan Gupta, Jiangzhuo Chen, and Madhav Marathe. 2020. Wisdom of the ensemble: Improving consistency of deep learning models. Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. 2019. Improving back-translation with uncertainty-based confidence estimation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 791–802, Hong Kong, China. Association for Computational Linguistics. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing Multiple Choice Science Questions. In Proceedings of the 3rd Workshop on Noisy Usergenerated Text, pages 94–106, Copenhagen, Denmark. Association for Computational Linguistics. Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, V. G. Vinod Vydiswaran, and Hao Ma. 2022. Idpg: An instance-dependent prompt generation method. In North American Chapter of the Association for Computational Linguistics. Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and LouisPhilippe Morency. 2022. Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation for consistency training. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating Generated Text as Text Generation. In Advances in Neural Information Processing Systems, volume 34, pages 27263–27277. Curran Associates, Inc. Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. 2019. S4l: Self-supervised semisupervised learning. Yue Zhang, Hongliang Fei, Dingcheng Li, and Ping Li. 2022. PromptGen: Automatically generate prompts using generative models. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 30–37, Seattle, United States. Association for Computational Linguistics. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017–5033, Online. Association for Computational Linguistics. Chunting Zhou, Junxian He, Xuezhe Ma, Taylor BergKirkpatrick, and Graham Neubig. 2022a. Prompt consistency for zero-shot task generalization. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022b. Large language models are human-level prompt engineers. A Generating Predictions from GPT-3
The GPT-3 API does not allow direct access to all the token probabilities, and the method of gathering logits through multiple API calls for each answer choice is cost-prohibitive. In order to perform rank scoring with GPT-3, we generate the best answer from the model by asking for deterministic generations (temperature of 1) and using 
 as the stop token. We take the log probability as the sum of the token log probabilities up to and including the first 
 token. While GPT-3 generally does well at following directions, it often does not generate an answer which exactly matches one of the multiple-choice options. We map each GPT-3 generation to one of the valid options by finding the answer that has the greatest 1,2-gram overlap with the generation (after lowercasing and removing punctuation and whitespace). We label a GPT-3 response as invalid if it has no overlap with a valid option. When evaluating confidence estimates, we filter out instances that resulted in at least one invalid prediction for a prompt. See Table 10 for statistics about the number of valid GPT-3 predictions. B Dataset Information
We present information about the datasets in Table 10: links to access to datasets on Hugging Face, the size of validation split, the number of instances that GPT-3 generated valid predictions for on the official prompt, and the number of instances that GPT-3 generated valid predictions for across all Multiple Human prompts. C Number of Prompts Per Dataset
Table 3 shows the number of Multiple Human (MH) prompts and automatically generated prompts (AGP) per dataset. In addition to these prompts, each dataset has a single prompt (Official Prompt) which comes from the paper in which the dataset authors introduced the dataset. D Confidence and Accuracy per Dataset
D. 1 Confidence
Table 4 shows the ECE and swapped pairs results for each dataset when using human-written prompts. Table 5 shows the ECE and swapped pairs results for each dataset when using automaticallygenerated prompts. D. 2 Accuracy Table 6 shows the accuracy results for each dataset when using human-written prompts. Table 7 shows the accuracy results for each dataset when using automatically generated prompts. E Automatically Generated Prompts
E. 1 Instructions Used for Prompt Generation In Table 9, we list the instructions that were used to generate additional prompts. These instructions come from the prompts used to train T0 (Sanh et al. , 2021). E. 2 Prompt Generation Prompts In Table 8, we list the prompt generation prompts (PGP) that were used generate new prompts. Within each PGP, we substitute an instruction from Table 9 in place of “{{ instruction }}” before gathering a response from GPT-3. E. 3 Paraphrased Instructions In Table 11, we provide the number of paraphrased instructions per dataset. We include statistics about the total number of unique paraphrased instructions and the final number of paraphrased prompts (after randomly selecting up to 50 prompts per dataset). In Tables 12 to 21 we provide the paraphrased instructions for each dataset. Table 8: Prompt Generation Prompts that are fed to GPT-3 in order to generate prompts. The PGP in row 1 is taken from Zhou et al. (2022b). ID Prompt Generation Prompt
1 Generate a variation of the following instruction while keeping the semantic meaning. Input: {{ instructions }} Output:
2 What’s another way of saying "{{ instructions }}" while keeping the same semantic meaning? Output:
3 Rephrase the following instructions while keeping the same semantic meaning. Input: {{ instructions }} Output:
4
Can you tell me another way of saying the following instructions while keeping the semantic meaning? Input: {{ instructions }} Output:
5 Paraphrase the following instructions while keeping the same semantic meaning. Input: {{ instructions }} Output:
6 Tell me another way of stating "{{ instructions }}" while keeping the same semantic meaning. Output:
7 How can I rephrase the instructions "{{ instructions }}" while keeping the same semantic meaning? Output:
8 Give me a sentence that expresses the following instructions in different words. Input: {{ instructions }} Output:
9 Generate a variation of the following instruction. Input: {{ instructions }} Output:
10 What’s another way of saying "{{ instructions }}"? Output:
11 Rephrase the following instructions. Input:{{ instructions }} Output:
12 Can you tell me another way of saying the following instructions? Input: {{ instructions }} Output:
13 Paraphrase the following instructions. Input: {{ instructions }} Output:
14 Tell me another way of stating "{{ instructions }}". Output:
Continued on next page
Table 8 – continued from previous page ID Prompt Generation Prompt
15 How can I rephrase the instructions "{{ instructions }}"? Output:
Table 9: Instructions from T0 prompts (Sanh et al. , 2021) that were used to generate new prompts. Dataset Instruction CoS-E v1. 11 Pick the option in line with common sense to answer the question. CoS-E v1. 11 Choose the most suitable option to answer the above question. CoS-E v1. 11 The best answer is
Cosmos QA According to the above context, choose the best option to answer the following question. Cosmos QA According to the above context, answer the following question. Cosmos QA Pick the best answer from the following options Cosmos QA Read the following context and choose the best option to answer the question. Cosmos QA Read the following context and answer the question. DREAM Read the following conversation and answer the question. QASC Given the two facts above, answer the question with the following options:
QASC You are presented with the question and the following answer choices. Now knowing the facts, choose the best answer. QASC You are presented with the quiz. But you don’t know the answer, so you turn to your teacher to ask for hints. He says the following facts. So, what’s the best answer to the question?
Quail According to the above context, choose the correct option to answer the following question. Quail The correct answer is Quail Pick the correct answer from the following options Quail Read the following context and choose the correct option to answer the question. Quarel Choose between “X” and “Y”. Quarel Do not use A and B to answer the question but instead, choose between “X” and “Y”. Quarel What is the most sensical answer between “X” and “Y”? Quarel Choose the answer between “X” and “Y”. Quarel I am testing my students’ logic. What is the answer they should choose between “X” and “Y”?
Quartz Answer the question based on the following text. Quartz Answer the question below Quartz Given the facts below, answer the question Quartz Having read the above passage, choose the right answer to the following question Quartz Read the passage below and choose the right answer to the following question Quartz Use information from the paragraph to answer the question. SciQ Answer the following question given this paragraph SciQ Read this paragraph and choose the correct option from the provided answers:
Social IQA Which one of these answers best answers the question according to the context?
WIQA How does the supposed perturbation influence the second effect mentioned? Answer by more, less or no effect. Table 12: Automatically generated instructions for CoS-E v1. 11. ID Instruction 1 Choose the option that makes the most sense to answer the question. 2 Choose the most logical answer to the question. 3 Choose the most practical option to answer the question. 4 Choose the answer that makes the most sense. 5 Choose the option that best answers the question. 6 What is the best answer to the question above? 7 Select the best option to answer the question. 8 Select the option that best answers the question. 9 What is the best answer to the question? 10 Select the best answer for the question above. 11 Choose the best option to answer the question. 12 What is the best option to answer the question? 13 Please select the option that best answers the question. 14 Choose the best answer to the question above. 15 The most correct answer is 16 One possible answer is. 17 The most accurate answer is
18 The most accurate answer is, The most precise answer is
19 What is the best answer? 20 The most accurate answer is the one that is closest to the correct answer 21 Choose the option that most makes sense to answer the question. 22 Choose the most sensible option to answer the question. 23 What is the best option to answer the question above? 24 Pick the best option to answer the question. 25 Select the most appropriate response to the question above. 26 What is the most suitable option to answer the above question? 27 Please select the option which you believe best answers the question. 28 Pick the best answer for the question above. 29 What is the best response to the question? 30 The answer that is most advantageous/ beneficial/ favorable is the best answer 31 The most correct answer is the one that is closest to the answer key 32 The answer that is most accurate or precise is the best answer. 33 The most ideal answer is. 34 Choose the option that seems most reasonable to answer the question. 35 Choose the answer that makes the most sense given the question. 36 The most sensible answer to the question is the one you should choose. 37 Pick the option that you think makes the most sense to answer the question 38 The most logical answer to the question is the best option. 39 Please select the option which you believe is the most sensible answer to the 40 From the given options, select the one that best answers the question. 41 Select the option that best responds to the question. 42 From the options below, select the one that best responds to the question 43 There is more than one correct answer to the question. Please choose the 44 Pick the best option to respond to the question. 45 Choose the most appropriate option to answer the question. Continued on next page
Table 12 – continued from previous page ID Instruction 46 The most optimal answer is 47 Choose the answer that you think is most correct. 48 The most correct answer is the one that is most accurate and precise. 2 Read the following context and answer the question below. What does 3 Choose the option that best answers the question based on the context above. 4 Read the text below and answer the question that follows. 5 Choose the most correct answer from the following options. 6 What does the author say about the best option? 7 What is the author’s purpose in writing this text? 8 Which of the following is the best option to answer the question? 9 After reading the context, answer the question. 10 In light of the information provided, please answer the following question. 11 What is the main idea of the text? The main idea 12 Please choose the option that best answers the question. 13 Choose the option that best answers the question based on the information given. 14 Read the following text and choose the best option to answer the question. 15 Read the context and choose the best option to answer the question. 16 To complete this task, read the text and then choose the best answer
17 What is the most important advice from the text? The most 18 What is the best option to answer the following question, based on the 19 Please select the option which you think is correct, based on the context 20 Based on the information given, select the most appropriate response. 21 Please select the option that you believe best answers the question based on the
22 Assuming you want a similar phrase with different words: Please 23 Choose the most correct answer from the given choices. 24 What is the answer to the question, based on the context above? 25 Please read the following information and select the best option to answer the question 26 According to the context above, choose the best option to answer the following 27 In light of the information given, please answer the following question. 28 Choose the most suitable answer from the given choices. 29 What is the author’s opinion on the matter? 30 Please read the following information and answer the question that follows. 31 Choose the best answer from the following options. 32 Choose the most accurate answer from the given choices. 33 Based on the information given, answer the following question. 34 Read the text below and answer the question. 35 In light of the above information, select the most appropriate response to the 36 Please answer the following question given the context above. 37 Choose the most correct answer from the following choices. 38 What are the instructions asking you to do? Read the following
39 Read the text below and select the best answer to the question. 40 What does the author say about the relationship between the two countries? 41 What is the best answer from the following options?
Continued on next page
Table 13 – continued from previous page ID Instruction 42 What’s the best answer from the following options?
43 What is the main idea of the text? The text is 44 What is the best option? 45 Read the following context and select the best option to answer the question. 46 Choose the best option to answer the question based on the following context. 47 Please select the option that best answers the question based on the context above 48 Please read the following text and select the best answer to the question below 49 Please answer the following question based on the information given above. 50 Choose the best option to answer the question based on the context provided. 3 Read the following conversation and answer the question below. Who is 4 Read the conversation below and answer the question. 5 What is the conversation about? What is the question about?
6 What is the conversation about? 7 Read the following conversation and then answer the question. 8 Please read the conversation below and answer the question that follows. 9 Read the conversation below and answer the question. Who is the
10 Read the following conversation and answer the question. 11 Read the following conversation and answer the question. Who is speaking
12 What is the conversation about? What is the main topic of
13 What does the conversation below reveal about the speaker? Read the
14 Read the following conversation and answer the question below. Two friends 15 Please read the conversation and answer the question. 16 Read the following conversation and answer the question. Who is the 17 Read the conversation and answer the question. 18 What is the next line in the conversation?
19 Read the conversation below and answer the question. At what time
Table 15: Automatically generated instructions for QASC. ID Instruction 1 You are given the quiz, but you are unsure of the answers. 2 Based on the information provided, please select one of the following options: 3 You are presented with a quiz, and you don’t know the answer 4 You are given a question and the following answer choices. Choose the best 5 Choose the best answer from the given choices, based on the information given 6 Choose the best answer from the given choices that best aligns with the 7 The teacher said that the answer to the question is one of the following 8 Taking into account the two facts mentioned above, please select one of the 9 Given the two facts above, please answer the question with one of the 10 1. Given the two facts, answer the question with the following options 11 You are taking a quiz and you don’t know the answer to 12 What is the best way to answer the question given the two facts?
13 Choose one of the following options: - A - B 14 You are given the quiz, but you are unsure of the answer. 15 Choose one of the following options based on the two facts given above. 16 Choose one of the following options that best answers the question: 17 Choose the answer that best fits the question, based on the information given 18 Assuming that the average person sleeps eight hours a day, how long will
19 The teacher provides you with the following information: You are presented
20 Choose one of the following answers: A) The moon orbits
21 Based on the information given, select one of the following options: 22 Assuming the two facts are true, which of the following is most likely 23 Based on the two facts provided, please select from the following options to 24 You are given the question and the following answer choices. With the information 25 Assuming the two facts above, answer the question with the following options: 26 You are taking a quiz and don’t know the answer to one of
27 What is the most likely explanation for the data? -The 28 Given the question and the following answer choices, select the most accurate answer
29 What is the probability that the person is a Democrat? What 30 The teacher provides you with the following information to help you answer the question 31 You are given a quiz, but you are unsure of the answers. 32 You are given a quiz, but you don’t know the answer. 33 You are presented with a quiz, but you are unsure of the answer 34 Read the question and the answer choices carefully, then select the most correct 35 Assuming the aforementioned facts are accurate, please select from the following options: 36 What is the probability that the box contains a white ball?
37 What is the result of subtracting 4 from 9? - 38 You are taking a quiz and are unsure of the answer to one of 39 Choose the best answer from the given choices. Continued on next page
Table 15 – continued from previous page ID Instruction 40 Choose one of the following options that best answers the question based on the 41 You are presented with the quiz. But you don’t know the answer
42 Choose one of the following options: a) The moon orbits
43 Given the question and the following answer choices, select the most accurate response 44 What’s the best answer to the question, given the following facts? 45 Choose the best answer from the given choices that best fits the question. 46 Choose the answer that best fits the question based on the given information. 47 Choose the answer that best fits the question, based on the given information 48 You are given the question and the following answer choices