Are Language Models Worse than Humans at Following Prompts? It's Complicated
Authors: Albert Webson
Alyssa Marie Loo
Qinan Yu
Ellie Pavlick
Venue: EMNLP 2023 Findings (2023)
Keywords: instruction following
prompting
human study
natural language inference

Abstract: Prompts have been the center of progress in advancing language models' zero-shot and few-shot performance. However, recent work finds that models can perform surprisingly well when given intentionally irrelevant or misleading prompts. Such results may be interpreted as evidence that model behavior is not "human like". In this study, we challenge a central assumption in such work: that humans would perform badly when given pathological instructions. We find that humans are able to reliably ignore irrelevant instructions and thus, like models, perform well on the underlying task despite an apparent lack of signal regarding the task they are being asked to do. However, when given deliberately misleading instructions, humans follow the instructions faithfully, whereas models do not. Our findings caution that future research should not idealize human behaviors as a monolith and should not train or evaluate models to mimic assumptions about these behaviors without first validating humans’ behaviors empirically.

Are Language Models Worse than Humans at Following Prompts? It's Complicated
Albert Webson "d Alyssa Marie Loo Qinan Yu" and Ellie Pavlick
*Department of Computer Science Program in Linguistics
Department of Philosophy
Brown University
Abstract
used by Webson and Pavlick (2022) (W&P here- after) to study how instructions in prompts affect LMs' performance. W&P manually write a set prompts for various natural language inference (NLI) and coreference resolution datasets. These prompts cover three main categories: instructive, irrelevant, and misleading. For example, Given [sentencel) can assume it is true that (sentence2) is an instruc- tive prompt for NLI, whereas (sentencel) Does the above passage express a positive sentiment? (sentence2) is a misleading instruction for NLI. (See Table for full definitions and examples.)
Prompts have been the center of progress in advancing language models' zero-shot and few- shot performance. However, recent work finds that models can perform surprisingly well when given intentionally irrelevant or misleading prompts. Such results may be interpreted as ev- idence that model behavior is not "human like". In this study, challenge central assumption in such work: that humans would perform badly when given pathological instructions. We find that humans are able to reliably ignore irrele- vant instructions and thus, like models, perform well the underlying task despite an apparent lack of signal regarding the task they are being asked to do. However, when given deliberately misleading instructions, humans follow the in- structions faithfully, whereas models do not. Our findings caution that future research should not idealize human behaviors monolith and should not train or evaluate models to mimic assumptions about these behaviors without first validating humans' behaviors empirically.
W&P assume that, if models perform held-out tasks by reading prompts instructions in the way that humans (are assumed to) do, their performance with instructive prompts should be much higher than their performance with other pathological cat- egories of prompts, namely:
instructive > mislcading
(A1)
instructive > irrelevant
(A2)
1 Introduction
instructive > null (no instruction)
(A3)
Prompting has emerged as the default way of using large language models (Brown et al., 2020; Sanh et al., 2022; Wei et al., 2021; Ouyang et al., 2022; Chung et al., 2022). However, collection of re- cent papers show that models perform surprisingly well when given misleading or irrelevant prompts (Webson and Pavlick, 2022: Prasad al., 2022; Khashabi et al., 2021), corrupted in-context exam- ples (Min et al., 2022), or corrupted explanations or chain-of- th-ouught (Madaan and Yazdanbakhsh, 2022; Ye and Durrett, 2022). Such results raise questions about whether language models' ability to follow instructions is analogous to humans" abil- ity to do so.
Instead, W&P find that T5 (LM-Adapted, 11B, Lester et al., 2021), TO (11B, Sanh et al., 2022), and GPT-3 (175B, Brown et al., 2020) do not exhibit the above patterns. Rather, in both zero-shot and few- shot settings, models perform roughly the same on instructive, misleading, and irrelevant prompts, violating A and A2 above. Models do, however. perform better given any type of instructions than they do with no instructions (i.e., A3 holds). There- fore, W&P conclude that while prompts do confer substantial empirical benefits, the fact that models so good at inferring the gold labels under vari- ous pathological prompts casts doubts on whether models understand or use instructions in ways sim- ilar to how humans do.
In this paper, we investigate what humans do in such settings. We follow the experimental setup
In this paper, we revisit W&P's assumptions on how humans behave with pathological prompts.
"Fqual contribution. Correspondence to papers@aw.nyy

---

Category
Description
Examples
How would describe the NLT lask human who has never the lask before.
[senience1] Are we justified in saying that "[sentence2]"" "??
instructive
Given [senlencel] 1] Should ussume that "[sentence2)" "i true?
Instruct the models to perform a task unrelated
|sentencel] / is the sentiment positive? [sentence2] [senience1] is this a sports news? [senience2]
misleading
to NLI.
Concatenate premise, a sentence unrelated to any NLP task, and the hypothesis.
[sentencel] bonito flakes boil more than few scconds the stock becomes too strong. "(sentence2)"2
irrelevant
Concatenate premise and hypothesis without any additional text.
[premise] (hypothesis) [sentence2] (sentencel)
null
Table 1: Prompt categories adapted from W&P, with W&P's two misleading categories collapsed into one 'mislead- ing" category for clarity. See Table for the full list.
We use the same experimental design but adapt for measuring human behaviors. In the zero- -hot setting, we find that while assumptions Al and A3 are consistent with human behaviors, A2 is not.
Crucially, when the instruction is misleading, we manually select examples such that <sentl, mi misledingg instruction, sent2> and <sentl, NLI instruction, sent2> al- ways have opposite gold labels-see Figure 1. As- suming participants are competent at NLI as well as at each of the relevant surface tasks² used in our experiments, this design enables to distinguish whether the participant is performing the NLI task or surface task when given misleading instructions.
Our experiments underscore the importance of validating assumptions about human behavior natural language tasks since, frequently, re- searchers' intuitions about human behavior do not bear out in practice, and that extra care should be taken in designing a fair comparison between mod- els and humans (Pavlick and Kwiatkowski, 2019; Dasgupta et al., 2022: Lampinen, 2022).
Procedure To ensure this experiment is as zero- shot as possible, each participant receives only one test item, followed by four additional items which we use as controls to ensure that all tasks and exam- ples are fair (Appendix E). For each item, subjects choose between" Yes" or "No". They do not receive any feedback throughout the experiment.
2 Experiment
Overview Following W&P, we use natural lan- guage inference (NI.I) as the primary task for our experiments. (That is, in our results, we always re- port human and model performance with respect to the NLI task.) When necessary, in designing stimuli for the misleading prompt condition (discussed in detail below), we additionally draw examples from 7 other tasks: lexical overlap, lexical identity, para- phrasing identification, grammatical acceptability, summarization acceptability, topic classification, and language identification (see Appendices H.2 and H.3 for details); we refer to these collectively as surface tasks in this paper.
Example Selection Because main goal is measure the effect of instructions and not humans' performance on the tasks themselves per se, we manually select examples that are as easy and un- ambiguous as possible. For the instructive, irrele- vant, and null conditions. we choose examples from RTE (Dagan et al., 2006) and MNLI (Williams et al., 2018). For the misleading condition, man- ually select all examples to ensure that the NLI labels differ from the misleading task labels. A full description of how we curate examples detailed in Appendix
We define an example to be pair of <sentencel, sentence2>, For NLI, sen- tences and 2 are the premise and hypoth- esis, respectively.' We define an item as a unique 3-tuple <sentencel, instruction, sentence2>, i.c., an example fitted within a prompt template, which can be instructive (w.r.t. NLJ), misleading, irrelevant, or empty (null).
Instruction Templates We select and lightly edit 22 of the 27 prompts used in W&P for testing, The complete prompts are listed in Appendix H. Combined with examples, there are a total of 194 unique items in our test condition (Table 2). Each item is assigned to minimum of three annotators.
For other tasks, they do not need sentence pair (e.g.,
sentiment analysis), our instructions always unanbiguously ask for judgment sentence!
We conducted our study on Ama- zon Mechanical Turk, receiving total of 597 re-
Participants
AW&P further differentiate moderately misleuding from extremely misleading instructions. However, for our purposes, we collapse this distinction, except where discussed in $A.2.
Seee Appendix E for a detailed discussion and analysis of control conditions which we take to be convincing evidence

---

Christopher Reeve, an uctor and director who becume inspiration worldwide after being paralyzed in a horse riding accident, died Sunday heart failure.
A male rabbic called buck and fomale rabbit is called doe, like deer.
Soprano's Square: Milan, Iialy, home the famed La Scala opera house. hanored soprano Maria Wodnesday when remsmee a new square after the diva.
Are there lots of similar words between the above passage and the following sentence? Female rabbit is called buck?
Single-Samily zoning is bad "La Scala opera house is located in Milan, Italy"?
Question: Docs this imply that, Christopher Reeve had an accident.
Yes
Yes No
Yes
No
[Inst. w.t.t. NLE: Instrnetive
{lost. w.r.t. NLI: Misleading
(Inst. w.r.. NLI: Irrelevant
Suface instruction: NLI
Surface instruction: Lexical Overlap
Surface instruction: None
Surface label: NiA
Surface label: Yes
Surface label: Yos
label: Yes)
NLI label: Yes)
NLJ label: No)
(c) Ireeevant/Null Cond.: No action- ahle task, only distractor (for irre-- evant) empty string (for null). Above shows the irrelevant condition.
(a) Instructive Condition: Baseline where the surface instruction is the same NLI.
(b) Misleading Condition: Surface task label always differs from the NLI label.
Figure Main experimental design. Note that under the misleading condition, a participant interprets the surface instruction as lexical overlap, then the gold answer would be "yes". If participant somehow interprets this instruction as NLI, then the gold answer will be "no". Text within the curly bracket are not shown to the participants.
Category
Prompts
Examples
Total Items
models. The overall performance of TO++, Flan-T5 and GPTs by themselves is consistent with the model performance reported by W&P. But, with the exception of GPT-4, models show very different patterns from that of humans when given misleading prompts. As expected, when humans are explicitly asked to do a task other than NLI (e.g., sentiment analysis, grammaticality), they tend to do the specified task (leading to low accuracy when measured against the NLI task). In contrast, models often appear to behave as though they have been instructed to do NLI even though they are instructed to do some other surface tasks (leading to high accuracy when measured against the NLI task). For example, when given misleading instruction (e.g.. paraphrasing identification opposed to NLI) (sentencel) Does that have the same meaning as " "(sentence2) ) "?, humans do indeed perform the paraphrasing task and thus receive a score of 0 on NLI, whereas models tend towards performing NLI, with T0++/GPT-4 receiving an NLI score of 1 and Flan-T5/GPT-33 a score of 0.6. (Full results on all prompts in SG.1.) This pattern confirms W&P's assumption A1 that models perform 'too well' on misleading prompts, i.e., better than humans would under similar conditions.
Instructive 5
12
60
Misleading
10
5
50
Irrelevant 5
12
60
Null 2
24
Total
22
46
194
Table 2: We define an item as a unique 3-tuple <sentencel, instruction ion, sentence2> where each sentence pair is an example manually selected from a dataset.
sponses over a three-day span. Participants were paid base rate of $0.50 with an additional $0.10 per correct answer as an incentive. See Appendix E for details on how we qualify participants.
Models To compare human performance with that of models, we use instruction -uunod mod- els T0++ (1IB, Sanh et al., 2022) and Flan-T5 (IIB, Chung et al., 2022), well as GPT-3.5 and GPT-4 (June 2023 ver- sion, OpenAl 2023). T0++ and Flan-T5 models are extensively fine-tuned to follow NLP instructions, with the key difference that TO has NLI as a held- out task while Flan-T5 does not. All models are given test items identical to those from the human experiments. (See $E.4 for additional details).
31pt-3.5-turbuy
3 Results
Figure/ 2 show the zero-shot performance of humans compared to that of instruction-tuned
However, when consider assumption A2 (instructive irrelevant), we see different story. When given irrelevant instructions, we see that all models and humans exhibit similar patterns. In fact, humans show far less variance than models in performing the NLI task when
that our participants are sufficiently competent.
"We add a line break after the premise text for better human readability, and edit all prompts to be clearer by making use of the line break (c.g. "the ahove passage").

---

0.8
08/055
NLI Accuracy
0:718
666667
0.6
0.4
0.2
0
T0
FLAN-T5
GPT-3.5
GPT-4
Humans
instructive
misleading
inrelevant
null
Figure 2: Zero-shot accuracy of human annotators vs. instruction-tunee models. In the misleading condition, lower NLI accuracy is preferred as it means higher accuracy in following the instructions to perform the surface tasks. Medians are displayed the bars. Each scatter point represents the accuracy on instruction within semantic category. For models, this is calculated as the mean accuracy over all the test items constructed from the examples for the instruction (i.e., 5 for each misleading instruction, 12 for each null, irrelevant and instructive instruction); for humans this the mean accuracy over all participants who received items with that instruction.
the instructions are irrelevant, suggesting that humans are more likely to perform NLI as some kind "default" task absent of useful instructions. For example, given the irrelevant instruction (sentencel) Inflect ions are annoying and thank god that
dency to do the same with irrelevant prompts is likely more feature than bug (cf. Merrill et al., 2022).
Such idiosyncrasies in human behavior are of- ten difficult to anticipate and even more difficult to codify in a way that lends itself well to bench- marks. Thus, Callison Callison-Burch Burch, 2016; Ross and Pavlick, 2019; Pavlick and Kwiatkowski, 2019; Dasgupta et al., 2022; Lampinen, 2022) in emphasizing the diffi- culty of evaluating models when "human-likeness" is the intended gold standard. As NLP models be- come increasingly advanced, and evaluation tasks increasingly complex, we are likely to face increas- ing challenges in determining whether models' be- havior is "aligned" with that of humans. This study contributes the line of work which underscores the importance of empirically measuring, rather than presupposing, human behavior such set- tings, as humans in practice routinely evade basic intuitions. Appendix B further discusses how to design a fairer comparison between humans and models.
Middle English got rid of most of them. (sentence2) Humans, TO++, Flan-T5, GPT-3.5 and GPT-4 all score similarly at 0.79, 0.83, 0.83, 0.67 and 0.75 respectively ($G.1).
echo recent work (Pavlick and
4 Discussion
Our findings show that, in zero-shot setting, hu- uppear largely faithful to prompt instructions. They perform well given instructive prompts and poorly given misleading prompts (as expected). However, we observe that TO++, FLAN-T5, and (to a lesser extent) GPT-3.5 are inclined to perform NLI in a zero-shot setting regardless of what is in fact being instructed. GPT-4, however, does seem to exhibit more human pattern in following the misleading instructions (albeit with high variance), It is possible that combination of pretraining FLOPs, fine-tuning data quality, and RLHF may have bridged some discrepancies between GPT and human behaviors, whereas smaller and supervised nne-tuning-only models fail to do so, but is im- possible to conclude given that little is known about GPT-3.5/4's s technical details.
5 Conclusion
In this work, we measure human behaviors when given misleading and irrelevant instructions for var- ious NLP tasks. We show that our prior work used oversimplified assumptions of human behavior to evaluate NLP models. Our results underscore the need to empirically validate assumptions about hu- man behavior, which often more complex in ality than our intuitions would lead to believe.
When no useful instructions are provided in the irrelevant prompt setting, our results show that hu- mans still tend to perform the NLI task surprisingly well. Contrary to W&P's criticism, models' ten-

---

6 Limitations and Future Work
this work. We thank Brendan Ho who wrote a lot of feedback for the free-form response questions. Special thanks to Roman Feiman for discussions on the design of the human experiments.
Our main experiment investigates only the zero- shot setting in humans. We do run pilot exper- iment with the intention to compare model and human behavior in a few-shot setting, reported in Appendix A. While the pilot already yielded inter- esting results, we leave a full experiment on the few-shot setting to future work.
References
Tom B. Brown, Benjamin Mann, Nick Ryder, Melamie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, lly Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Our experimental design also only uses the Natu- ral Language Inference (NLJ) task as the reference task, which we acknowledge may be a task that is perhaps more intuitive as a "default" task than other common NLP tasks (c.g., sentiment analy- sis, paraphrase). While future work should con- sider repeating this analysis for other tasks, NLI has several advantages for our purposes. First, NLJ is one of the only tasks explicitly held out entirely from the instruction-tuner TO models, allowing for the best evaluation of their few-shot performance. Moreover, the intuitiveness of NLI works in our favor for the claims we make-it would be ostensi- bly most challenging for humans to override a bias towards this task to instead follow task instructions. Then, the fact humans do this even under the NLI task setting is the strongest evidence of instruction- following behavior compared to any other task. We acknowledge that there is future work to verify that the human behavior observed for instruction- following extends to other tasks.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu. Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Fdd H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts. Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction finetuned ArXiv preprint, abs/2210. 1141..
instuuctionfiiettueed language models.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- wral Language Processing, pages 2475-2489 Brus- sels, Belgium. Association for Computational Lin- guistics.
7 Ethics Statement
We acknowledge that calling for more rigorous hu- man benchmarking only exacerbates NLP field's needs for human annotators, where it has been demonstrated that NLP crowdsourcing may po- tentially expose workers to harm, as described in Shmueli et al. (2021). For our study, our Institu- tional Review Board (IRB) reviewed our experi- mental design and determined that its primary aim is to study computational language models and thus does not meet the federal definition of human sub- jects research. Future studies should similarly sub- mit their studies, involving human benchmarking, for review by their institutions' IRBs.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment chal- lenge. In Machine learning challenges workshop. pages 177-190. Springer.
Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2022. Lan- guage models show numan-like content effects on reasoning. ArXiv preprint, bbs/2207.07051.
Philippe Goldammer, Hubert Annen, Peter Lucas Stöckli, and Klaus Jonas 2020. Careless responding in questionnaire measures: Detection, impact, and remedies. The Leadership Quarterly, 31(4):101384 3844
8 Acknowledgments
We thank Tal Lizen, Andrew Lampinen, Swaroop Mishra, Apoorv Agarwal, Michael Lepori, Louis Castricato, Samuel Musker, Aaron Traylor, and Philip LaDuca for discussions and comments on
Hchaen Robert Greszki, Marco Meyer, and Harald Schoen. 2015. Exploring the Effects of Removing "Too Fast" Re- sponses and Respondents from Web Surveys. Public Opinion Quarterly, 79.

---

Joel Jang, Seonghyeon Ye, and Minjoon Seo. 2022. Can large language models truly understand prompts? a case study with negated prompts. ArXiv preprint, 1ts/2209.12711.
OpenAI. 2023. GPT-4 Technical Report.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens. Amanda Askell, Peter Welinder. Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv preprint, abs/2203.02155
Daniel Khashabi, Shane Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sameer Singh, Sean Welleck, Han- nanch Hajishirzi, Tushar Khot, Ashish Sabharwal, et al. 2021. Prompt way waywardness: The curious case of discretized interpretation of continuous prompts. ArXiv preprint, abs/2112.06 12.08348.
Ellie Pavlick and Chris Callison-Buis Burch. 2016. So-called non-suhsective adjectives. In Proceedings of the Fifth Joint Conference on Lexical and Computational Se- mantics, pages 114-119, Berlin, Germany, Associa- tion for Computational Linguisties.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. ArXiv preprint, abs/2205.11912
Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. 2022. Can language models learn from explanations in context? ArXiv preprint. abs/2204.02329..
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transuc- tions of the Association for Computational Linguis- tics, 7:677-694
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2022. Grips: Gradient-free, edit-based in- struction search for prompting large language models. ArXiv preprint, abs/2203.07288
Andrew Kyle Lampinen. 2022. Can language models handle recursively nested grammatical structures? a case study on comparing models and humans. ArXiv preprint, abs/2210.11 15.103..
Alexis Ross and Ellie Pavlick. 2019. How well do NLI models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-ICCNP)), pages 2230-2240, Hong Kong, China. Association for Computational Linguistics.
Brian Lester. Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In EMNLP.
Aman Madaan and Amir Yazzanbakkshh. 2022. Text and patterns: For effective chain of thought, it takes two to lango. ArXiv preprint, 0w//2229.06688
Subhro Roy and Dan Roth. 2015. Solving general arith- metic word problems. In Proceedings of the 2015 Conference Empirical Methods in Natural Lan- guage Processing, pages 1743-1752, Lisbon, Portu- gal. Association for Computational Linguistics.
William Merrill, Alex Warstadt, and Tal Linzen. 2022. Entailment semantics can be extracted from an ideal language model. ArXiv preprint, abs/2209. 12407.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in-context learning work? ArXiv preprint, hbs/2202.11837.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Mattco Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2021. Natural instructions: Benchmarking generalization to new tasks from natural language instructions. ArXiv preprint, abs/2104.08773.
Rishabh Misra. 2022. News category dataset. arXiv preprint 2rXiv:2209.11423
Rishabh Misra and Jigyasa Grover. 2021. Sculpting Data for ML: The first act of Machine Learning.
Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-We Mei Ku. 2021. Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber. David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. 2021. Show your work: Scratch- pads for intermediate computation with language models. ArXiv preprimt, abs/21 12.001
Aarohi Srivastava, Abhinay Rastogi, Abhishek B Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta,

---

Adrià Garriga-Alonso Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Annasaheb Rahane, Anantharaman S. Tyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmuller, Andrew M. Dai, Andrew D. La, An- drew Kyle Lampinen, Andy Zou, Angela Jiang, An- gelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tahassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sab- harwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakacs, Bridget R. Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Ozyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Stephen Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, C'esar Ferri Ram írez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Buree
Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua. Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Ochieng' Omondi, Kory Wallace Mathewson, Kristen Chia- fullo, Ksenia Shkaruta, Kumar Shridhar, Kyle Mc- Donell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras- Ochando, Louis-Philiipe Morency, Luca Moschella, Luca Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Col'on, Luke Metz. Lutfi Kerem eSenel, Maarten Bosma, Maarten Sap. Maartje ter Hoeve, Madotto Andrea, Maheen Saleem Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, M Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew Leavitt. Matthias Hagen, M'aty' as Schubert, Medina Baitemirova, Melissa Arnaud. Melvin Andrew McElrath, Michael A. Yee, Michael Cohen, Mi Gu, Michael I. Ivanitskiy, Michael Star- ritt, Michael Strube, Michal Swkedrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Monica Tiwari, Mo- hit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, T Mukund Nanyun Peng. Nathan Chi. Naycon Lee, Neta Gur-Ari Krakover. Nicholas Cameron, Nicholas S. Roberts. Nicholas Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha Iyer, Noah Con- stant, Noah Fiedel. Nuan Wen, Oliver Zhang. Omar Agha, Omar Elbaghdadi, Omer Levy: Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung. Paul Pu Liang. Paul Vicol, Pegah Alipoormo- labashi, Peiyuan Liao, Percy Liang, Peter W. Chang, Peter Eckersley, Phu Mon Htut, Pi-Bei Hwang, P. Milkowski, Piyush S. Patil, Pouya Pezeshkpour, Priti Oli, Qiaozuu Mei, QING LYU, Qinlang Chen, Rabin Banjade, Rachel Eita Rudolph, Raefer Gabriel, Rahel Habacker, Ram's on Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand. Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Techan, Rylan Yang, Sahib J. Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruet- ter, Sam Bowman, Samuel S. Scheenholz. Sanghyun Han, Sanjeev Kwatra. Sarah A. Rous, Sarik Ghazar- ian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi S. Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Deb- nath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo hwan Lee, Spencer Bradley Torene, Sriharsha Hat- war, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Rose Biderman, Stephanie C. Lin, S. Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq A.
Chris Waites, Chris- tian Voigt, Christopher D. Manning, Christopher Potts, Cindy Tatiana Ramirez, Clara Rivera, Clemen- cia Siro, Colin Raffel, Courtney Asheraft, Cristina Garbacea, Damien Sileo, Daniel H Garrette. Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Gonz' alez, Danny Hernandez, Danqi Chen, Daphne Ippolito. Dar Gilboa, David Dohan, D. Drakard, David Jur- gens, Dehajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes. Diganta Misra, Dilyar Buzan, Dim- itri Coelho Mollo, Diyi Yang. Dong-Ho Ho Lec, Ekate- rina Shutova, Ekin Dogus Cubuk, Elad Segal. Elcanor Hagerman, Elizabeth Barnes, Elizabeth P. Donoway, Ellie Pavlick, Emanuele Rodolà, Emma FC Lam, Eric Chu. Eric Tang, Erkut Erdem, Emne Chang. Ethan A. Chi. Ethan Dyer, Ethan J. Jerzak, Ethan Kim, Funice Engefu Manyasi, Evgenii Zheltonozh- skii, Fan Xia, Fatemeh Siar, Fernando Mart" 'inez- Plumed, Francesca Happ François Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Paras- candolo. Giorgio Mariani, Gloria Wang, Gonzalo Simovitch-LLoppe. Gregor Betz, Guy Gur-Ari, Hana Galjjasevic, Han Sol Kim, Hannah Rashkin, Hanna Hajishirzi. Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hubert Wong, Ian Aik-Soon Ng. Isaac No- ble, Jaap Jumelet, Jack Geissinger, John Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández. Fisac, J. Brooker Simon, James Koppel, James Zheng, James Zou, Jan Koco'n n, Jana Thompson, Jared Ka- plan, Jarema Radom, Jascha Narain Sohl- Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jenni Marsh. Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Oluwadara Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Jane W Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jorg Frohberg, Jos Rozen, José Hemández--ralll, Joseph Boudeman, Joseph Jones,

---

Ali, Tatsuo Hashimoto, Te-Lin Wu, Theo Desbordes. Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, T. N. Kornev, Tim- othy Telleen-Lawton, Titus Tunduny, Tobias Ger- stenberg, Trenton Chang. Trishala Neeraj, Tushar Khot, Tyler O'Brien Shultz, Uri Shaham, Vedant Mísra, Vera Demherg, Victoria Nyamai, Vikas Rau- nak, Vinay Venkatesh Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fe- dus, William Saunders, William Zhang, W Vossen, Xiang Ren, Xiaoyu F Tong, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yang Song. Yasaman Bahri, Ye Ji Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yu Hou, Yushi Bai, Zachary Seid, Zhao Xinran, Zhuoye Zhao, Zi Fu Wang, Zijie J. Wang, Zirui Wang. Ziyi Wu, Sahib Singh, and Uri Shaham. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. ArXiv preprint, 0td//200.04611
formation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. ArXiv preprint, abs/2205.10625.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo- hananey, Wei Peng. Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguis- tie minimal pairs for English. Transactions the Association for Computational Linguistics, 8:377- 392.
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoe Le. 2021. Finetuned lan- guage models zero-shot learners. ArXiv preprint, abs/2109.01655
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le. and Denny Zhou. 2022. Chain of thought prompting elicits reasoning large language models. ArXiv preprint, abs/2201. 1903.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. Proceed- ings of the 2018 Conference of the North American Chapter of the Association, for Computational Lin- guistics: Human Language Technologies, Volume I (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Xi Ye and Greg Durrett. 2022. The unreliability of ex- planations in few-shot prompting for textual reason- ing. In Advances in Neural Information Processing Systems.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun, 2015. Character-level convolutional networks for text clas- sification. In Advances Neural Information Pro- cessing Systems 28: Annual Conference on Newral In-

---

A Pilot Few-Shot Experiment
Appendix
Our main experiment above focuses only on the zero-shot setting. However, model behavior (and likely human behavior well) can look very dif- ferent when given even single labeled example. Thus, we additionally conduct a pilot experiment to investigate human behaviors under few-shot learning setup. We consider two conditions. The first, few- seo- with labels, simulates the usual few- shot learning setting: the participant is given the gold label as feedback after answering each ques- tion. The second condition, nowssso few-shot without labels, is inspired by Min al. (2022)'s 's study in few-shot learning with corrupted labels. In this condition, we do not give participants any label feedback. How- ever, they still have the opportunity to refine their understanding of the instruction via sequential en- counters with different unlabeled examples.
Table of Contents
Pilot Few-Shot Experiment
A.1 Method
A.2 Results
10
B Reconciling Few-Shot and Zero-Shot
Evidence
11
Related Work
D Author Contributions
13
E Additional Details of the Main Zero-
Shot Experiment
13
E.1 Controls
13
A.1 Method
E.2 Qualifications
13
We show each subject 32 items sequentially, item per page, for which subjects must respond "Yes" or "No" each item. Each subject is assigned an instruction as condition and the instruction is the same across all examples they receive.
E.3
Instructions Workers
13
E,4
Evaluation Protocol Details
14
F
Additional Results for Zero-Shot Ex-
periment
15
F.I
Using Control Scores as Qualifi- cation
In the few-shot with labels condition, subjects are shown whether their answer was correct for that item. Here, correctness is always based on NLI (i.e.. whether sentence entails sentence 2) regardless of the surface instructions. Unlike the zero-shot study, there is no special design in the example sentence pairs in relation to the instruction, only that the sample is balanced in the number of entailments and non-entailments. That is, 50% score indicates random performance on the implicit NLI task.
15
F2 Without Response Time Qualifi- cation
15
G Additional Figures and Data for Zero- Shot Experiment
17
G.1 Accuracies Per Instruction
17
Instructions and Examples
18
Instructions
In the few-shot without labels condition, sub- jects do not receive any feedback throughout the 32 items. This setup inspired by Min al. (2022). who show that models can surprisingly well learn from corrupted labels. We conjecture that unsu- pervised exposure to examples may help humans pragmatically infer what researchers want out of them and what the underlying task is.
H.2 Example Sources for Test Con- dition
19
H.3 Example Sources for General Controls
20
I Results on Zero-Shot Experiment Con-
trol Conditions
21
L1 NLI Controls
21
I.2 General (Surface Task) Controls
post-Experiment Survey After participants complete the 32 items, we ask for paragraph-lenglt responses to two questions: (1) How did you decide to choose between "Yes" or "No"? (2) What did you think about the instructions we gave?
J Effect of Experience with Prior NLP Studies
23
Few-Shot Experiment Experiment Survey
Post-
25
Instruction Templates instruction from each of W&P's five categories for
We manually choose one

---

this pilot study:
The post-experiment survey also confirms that par- ticipants were able to figure out that the prompt was simply irrelevant (Appendix K).
1. [sentencel] Using only the above description and what you know about the world, "(sen- tence2]" definitely correct. Yes or no? (In- structive; adapted from MultiNLL's instruction to crowdworkers).
Paraphrasing Inst. (Misleading) When partic- ipant is given a paraphrasing instruction without label feedback, they were successfully misled perform paraphrasing identification, thus scoring much lower on NLI (dashed yellow line in Fig- ure 3). But when a participant given a paraphras- ing instruction with NLI labels as feedback, they quickly adapted their interpretation of the instruc- tion in order to fit the labels (solid yellow line in Figure 3). As one participant wrote in the post- experiment survey:
2. [sentencel] | Can the above be paraphrased as: "seettence))" (Moderately misleading.)
3. (sentencel) Is the next sentence grammati- cally correct? (sentence2) (Extremely mis- leading.)
4. (sentencel) Single-family zoning is bad for American cities [sentence2) (Irrelevant.)
In the first few questions, my strategy is to read through the entire paragraph sentence and then decide whether the paraphrased sentence makes sense not. However, then started look the paraphrased sentence first and decide whether it is correct or wrong based on the given piece of text.
5. [sentencel] ) (sentence2) (Null; empty instruc- tion baseline.)
Participants Subjects were undergraduate stu- dents (n = for the with labels condition, n = for the without labels condition). Subjects were asked to finish all 32 items in a single session within an hour. Subjects were paid base com- pensation of $15, with $0.25 bonus for every correct answer as an incentive. As it is expensive have a participant complete 32 items continu- ously in a single session (in order mimic models' few-shot training), and because the trend was suffi- ciently clear from our pilot experiment, we did not proceed with a larger pool of participants.
That is, this participant completely recovered the NLI instruction even though they were given a paraphrasing instruction. They even ob- served the unidirectional entailment nature of NLI (i.e., sentence2 does not need to entail sent sentencell the bidirectional entailment ture of paraphrasing (i.e., the two sentences must express upproximately the same meaning):
A.2 Results
Initially, also considered whether the paraphrased sentence captured all the ma- jor details or not, but the quiz later shows that comprehensiveness not a factor.
Figure shows the cumulative scores of subjects across 32 items are shown for both few-shot with labels and without labels conditions, where each line is one subject. Solid lines represent the perfor- mance of subjects with labels, and dotted lines rep- resent the performance of subjects without labels. The subject who received the instructive prompt achieved perfect score, and their performance is presented as green reference y xr line against all participants who received other prompts.
Grammaticality Inst. (Misleading) While para- phrasing is a task related to NLL, grammatical ac- ceptability is a much more misleading instruction since it has nothing to do with NLI. Here, we see the with and without label results nearly reversed: when given NLI label feedback, the grammatical acceptability instruction appears to be so incompat- ible with the labels that one participant was fused and unable adapt their interpretation just one task, and ultimately scored much lower on NLI (lower solid red line in Figure 3). In the post-experiment survey, they wrote
Irrelevant Instructions Participants who re- ceived the irrelevant instruction perform practically identically with or without label feedback. In both conditions subjects with the irrelevant prompt also performed almost just as well the subject who received the instructive prompt. Like the zero- shot study, this pattern provides evidence against W &'P's assumption A2 (instructive > irrelevant).
basically went on a gut reaction on what was correct. On some weird instances,

---

Paraphrasing (Misleading)
Crammaticality (Misleading)
Irrelevant
Null
25
NLI Accuracy
20
15
5
0
5
10
15
20
25
30
1
5
10
15
20
25
30
5
10
15
20
25
30
1
5
10
15
20
25
30
Trem Index
Few-Shot withour Labels
Few-Shot with Labels
Baseline with Instructive Prompt
Figure 3: Cumulative total scores between the few-shot with labels and few-shot without labels experiments, within categories. The y-axis plots the of correct answers out of z items answered far. The sequence of 32 prompts were the same across both experiments for all prompt categories. Each line is one subject.
thought hard about good grammar. Of- tentimes, looked the content of the sentence. If it was factually correct, guessed yes.
I thought that maybe grammar also compasses the validity of the statement based on the text, so chose my answers based on that.
However, another participant did eventually figure out the underlying NLI task:
Additional survey responses are included in Appendix K. Overall, human behavior in the single
few-shot cannot be readily summarized inequality as attempted in Al 3. Rather, different participants can respond to the same instructions in different ways; some interpret the instructions strictly and literally, while others adopt a more relaxed pragmatic interpretation, and still others refine their interpretations over time.
feel like the question should be changed because it seems like the question is actu- ally, "Is this statement true based on the context given the paragraph."
When no label feedback is giventhe results are more surprising and nuanced. Because all of our examples are handpicked and grammatical, participants found themselves answering "yes" to many examples and starting to question the
B Reconciling Few-Shot and Zero-Shot Evidence
instruction itselff
As we find instruction-tuned models seem to largely match human behaviors the few-shot set- ting, while falling short of human in the zero-shot setting, which setting should we weigh more for evaluating models' understanding of prompts in- structions? Concurrent work by Lampinen (2022) notes that zero-shot could be an inherently prob- lematic way to study the full competence of LMs: From a model's perspective, it has just finished imitating (for example, in T5's case) a trillion to- kens of highly heterogeneous content and linguistic styles with communicative intents far from answer- ing academic evaluations. In a zero-shot setting, it could be unclear to the model "what is the intended continuation; the model might be likely produce blank line for someone to fill in the answer, jokes mocking the question, or just arbitrary ram- bling" (Lampinen, 2022, 7); all of the above may be valid continuations for its language modeling
looked at whether the sentence was ac- curate to the information given in the text, and also if the sentence itself had cor- rect grammatical structure. It was little difficult because some of the sentences made inferences that weren't 't explicit in the given text, so wasn't sure that was grammatical error or not.
They then started incorporated the semantic well-formedness into their interpretation of grammatical acceptability:
Usually, think of something being grammatically correct when the sentence has correct grammatical structure, includ- ing punctuation and capitalization. Since most the sentences seemed to fit this,

---

pretraining objective. We partially address this is- sue by only using instruction-tune models, which are trained on large mixture of traditional NLP datasets and thus primed to directly answer the question.
lines perform poorly on an arithmetic dataset (Roy and Roth, 2015). Instead, W&P claim that the exis- tence of high-performing pathological prompts for a large number tasks show that they use prompts in an un-human-like way, while the existence of bad- bad-performing pathological prompts is orthogonal to this line of argument.
On the human side, in order to make fair com- parison, we carefully design our experiments to make sure the human responses are "as zero-shot as possible" by (1) having one participant answer only one test question and (2) having all qualifica- tion questions come after the test question so as to not bias the participants. This is highly controlled, perhaps even contrived, condition that does not re- flect well on how humans learn from instructions in the real world, even in most other cognitive science experiments where participants given familiarization trials prior the test trials.
Few-Shot Exemplars and Explanations Unlike zero-shot, the few-shot setting has more conflicting evidence in the literature on whether models per- form just as well with pathological prompts. Min et al. (2022) first showed that the correctness of the few-shot labels are not required, concluding that prompts are largely helping models to adapt to the domain and format of the input text as well as the space of possible labels.
often
Therefore, we agree with Lampinen (2022) that, for future studies, the few-shot setting is likely a more productive way to probe model's ' true com- petence, even though it may be scientifically less controlled in other respects, since now effects of the few-shot examples must be considered, which could also be counter-intuitive as shown in Ap- pendix A and Min al. (2022).
As few-shot exemplars are now commonly accompanied with intermediate computation, explanations, or chain-of-thoughts (Wei et al.. 2022; Nye et al., 2021; Zhou et al., 2022; Lampinen et al., 2022), Madaan and Yazdanbakhshh (2022) agree with Min al. (2022) and find that various few-shot chain-of-thought prompts-with corrupted symbols but retaining the overall task format-have performance comparable to non-corrupted baseline, thus arguing that "CoT helps a language model in imitating the prompt and generating the right tokens for the task-and is conceivably less related to their reasoning abilities." Similarly, Ye and Durrett (2022) find that explanations improve performance modestly for OPT and GPT-3 but improve substantially for text-davinci-002 on QA and NLI datasets. How- ever, the model-generated explanations themselves are often inconsistent or factually incorrect, con- cluding that "model internal 'reasoning' does not always align with explanations that it generates."
However, zero-shot evidence should not be ig- nored either, especially considering that there is consistent collection of work showing language models being insensitive instructions on tasks in addition to NLI and on models of various sizes and finettuning strategies.
Related Work
Zero-Shot Instructions In addition to W&P, many papers find similar results that models per- form well with semantically incoherent instructions on a variety of tasks. Prasad et al. (2022) find that semantically incoherent prompts work well for In- structGPT over 12 QA and coreference datasets in Natural Instructions Mishra et al. (2021), Khashabi al. (2021) find this to be true for GPT-2 on sentiment analysis and topic classification datasets Natural Instructions. Jang et al. (2022) show OPT and InstructGPT are unable to follow negated instructions over 9 QA and sentence completion datasets, performing well below human baseline of 13-ycar-olds.
With extensive statistical controls on a diverse range of BIG-Bench tasks (Srivastava et al., 2022), Lampinen et al. (2022) find that LMs' success with post-answer explanations do outperform baselines such as same-length word-scrambled explanations, domain-relevant but non-explanatory statements, and correct explanations misaligned with wrong few-shot examples. Notably, although they find a positive result with the effect of explanations, they also find that models are much more insensitive to the effect of instructions.
Note that none of the above papers, including W&P, claim that pathological prompts would per- form just as well for all tasks. Indeed, Kojima et al. (2022) show various irrelevant and misleading base-

---

D Author Contributions
We curate examples from range of datasets such that the misleading task and NLI task have op- posing answers, and also to be converse to the test condition. That is, if the misleading task answers for a prompt is "Yes" in the test condition, it would be "No" in the General Control. See Appendix H for how examples were selected.
Albert Webson led the project, co-designed the ex- periments, implemented the model evaluation code, and led the paper writing.
Alyssa Marie Loo co-designed the experiments, manually curated all prompts and examples, eval- uated all models, produced all analyses, and co- authored the paper.
NLI Control The NLI Control verifies if partic- ipants can perform the NLI task. In this control, subjects are given two items with an instructive prompt and are scored on performing the NLI task correctly.
Qinan Yu co-designed the experiments, con- ducted all human experiments, and co-authored the paper.
Ellie Pavlick advised the project and edited the paper.
E.2 Qualifications
E Additional Details of the Main
From the 597 responses, for our main results we exclude data from 93 subjects whose total com- pletion time t for the five-question study is less than one standard deviation from the mean of the sample (t < 33.01). Extremely low response times have been shown be an accurate indicator of care- less responding, where data from such "speeders" have been shown to lower data quality (Greszki et al., 2015; Goldammer et al., 2020). Past studies have typically defined floor cutoffs based on the distribution of their data, with no singular conven- tion. From data's response time distribution (Figure 4), there is a clear bimodal distribution between "speeder" response times and typical re- sponse times; floor of 33.01 sufficiently ex- cludes the "speeder" distribution to leave a sizeable sample of n = 504.
Zero-Shot Experiment
E.1
Controls
We collect additional data to test the robustness of our result on different subgroups of participants se- lected under multiple conditions. After the test con- dition, participants are asked to complete two addi- tional control conditions, General (Surface Tasks) Control and NLI Control, that provide such se- lection criteria for post-hoc analysis in Appendix These controls test whether the same trend still holds under more restrictions such as receiving per- feet scores on both controls as additional analysis. However, note that our main results do not exclude any participants using these controls as it may bias the results to only consider participants with spe- cific behavior patterns.
We also ran separate pilot study in order to determine whether previous exposure to other NL.P studies on Mechanical Turk would bias subjects to perform the underlying NLI task. The results were inconclusive and thus we decided not to exclude participants who had participated in NLP studies previously. See Appendix
The two controls are four items presented after the first item from the test condition, adding up to the five items that each subject is presented in the study. The control items are shown one by one in the following sequence: two items in the General Control, and then two items in the NL/ Control. As with the test condition item, subjects are asked to answer "Yes" "No" in response to each control item. The performance of our subject population on the controls is shown in Appendix I.
E.3 Instructions to Workers
Workers will see the following instructions before they begin the study.
General Control The General Control verifies if participants can perform the misleading tasks. In this control, subjects are scored on whether they perform the surface task correctly. If subjects were already given a misleading prompt in their condition, in this control they are shown the same prompt with two new examples. Otherwise, subjects are randomly assigned two items with a misleading prompt.
You will be given 5 short paragraphs on sepa- rate pages, each containing a yes-no question. You will be paid at least $0.5 in addition to an extra $0.1 for each correct answer.
If you have high number of correct an- swers, you will be qualified participate in
FEven if subject's 's data was excluded, they were still compensated.

---

00-33.01 62126622
Proportinn of participanis
0.008
0.006
0.004
100 200 300
400
500
600
700
800
900
Total response times (s)
Figure 4: Histogram of subjects' total time taken (in seconds) to complete the zero-shot study (n 597, μr 126.62, 93.62). The left dotted line is the floor cutoff (t 33.01): the 93 participants that had total response time lower than this cutoff were excluded from the main paper's analysis.
our full, $15 per hour study. You will not be graded based on how fast you finish.
You have 10 minutes to answer all 5 ques- tions, which should be plenty of time, but please complete our study one continuous session and do NOT navigate other tabs or tasks, as we are measuring the time takes you to respond to each question.
You can only work on this HIT once. Multi- ple submissions will be prevented.
E.4 Evaluation Protocol Details
Following Sanh et al. (2022) and Brown et al. (2020), we evaluate T0++ and Flan-T5 by rank classification. We evaluate GPT-3.5 and GPT-4 by string match, as it is longer possible get per- token token bog-pobbbbilities from OpenAl APls. Fortunately, GPTs are able follow instructions that return "Yes" or "No" as first token, so we use greedy decoding (temperature = 0) which is equivalent to single-token rank classification.
Each item is input into the GPTs as con- tent message in the user role, with the system prompt "You will be given short paragraph with a yes-no question. You strictly must answer only "Yes" or 'No' the paragraph.". ". No other instruc- tions or examples are given.

---

F
Additional Results for Zero-Shot Experiment
0.8
%
0.792
10:792
0.743
We present post-hoc analysis of our zero-shot ex- periment data using different subgroups of partic- ipants. For all figures in this section, medians are indicated in the bars. Each scatterpoint represents accuracy on a prompt within the semantic category, calculated as the aggregated accuracy over multi- ple annotators whose test item was constructed by the prompt and one of the prompt's five possible examples.
NLI Accuracy
0.6
0.4
0:286
0.2
0
F.1 Using Control Scores as Qualification
8 Instructive
misleading
inreevant E null
In Figure 5, Figure 6, Figure 7 and Figure 8 we present results using the controls as exclusion crite- ria.
Figure 5: Zero-shot accuracy data of human annotators with perfect IIJ Control scores and time above floor cutoff. (n 384)
Constraining data subjects that score perfectly on the General Control (Figure 6) decreases hu- man performance with misleading prompts on the NLI task in the test condition dramatically that is,
00.857
0.833
0.8
NLJ Accuracy
0.663
these subjects perform the non-NLI task extremely well right from the first item, given a prompt that instructs a misleading task. This sample likely se- lects for subjects that interpret instructions most strictly.
0.6
0.4
In contrast, results from constraining data sub- jects that score perfectly on the NLI Control (Fig- ure 5) remain highly similar to our main results. This supports that it is not incompetence with the NLI task that causes humans to score poorly in the test condition if given misleading prompt, but that they are indeed following the given instructions to perform the misleading task.
0.2
0
instructive
misleuding
irrelevant
null
Figure 6: Zero-shot accuracy data of human annotators with perfect General Control scores and time above floor cutoff. (n 238).
F.2 Without Response Time Qualification
In Figure 9, Figure 10, Figure 11, Figure 12 and Fig- ure 13 we present results without using response time qualification. Removing the response time qualification has the effect of introducing noise into the data: the overall variance between dif- ferent prompt categories reduces as all categories tend towards 0.50. However, the top-line trends we argue in the main paper about human behavior given these different prompts remain observable (instructive > misteading-moderate moderate: instructive misleading-extreme instructive ss irrelevant).
0.8
0.8
-0:81
0.769
NLI NLI. Accuracy
0.6
0.4
0.2
0.174
0
instructive
misleading
inrelevant null
Figure 7: Zero-shot accuracy data of human annotators with a total score of at least 3 out of 4 Control items and time above floor cutoff. (n 340).

---

10.909
0:989
0.8
0.833
0:824
0.8
NLI Accuracy
NLI NLI Accuracy
0.6
0.626
0.633
0.6
0.4
0.4
0.2
0.2
0
0
instructive
misleading
irrelevant
null
8 instructive
misleading
irrelevant m null
Figure 11: Zero-shot accuracy data of human annotators with perfect General Control scores, with any response time. (n 273).
Figure 8: Zero-shot accuracy data of human annotators with all perfect NLI and General Control scores and time above floor cutoff. (n 186).
00.86
0.8
0.773
0.729
NLI Accuracy
0.7
0.718
0.727
0.681
0.6
0.6
NLI Accuracy
0.4
0.4
0:261
0.366
0.2
0
instructive
misleading
irrelevant null
Zero-shot accuracy data of human annotators with total score of at least 3 out of 4 Control items. with any response time, (n 377).
Figure 12:
instructive
misleading
inrelevant null
Figure 9: Zero-shot accuracy data of human annotators with any response time (n 597) (i.e., data).
0.9
0.889
0.8
0.792
0.76
0.744
0.8
0.7
0.6
NLIAc Accuracy
NLI Accuracy
0.633
0.6
0.5
0.4
0.4
0.3
0.286
0.2
0.2
0.1
instructive
misleading
irrelevant null
instructive
misleading
inrelevant null
Figure 13: Zero-shot accuracy data of human annotators with all perfect NLI and General Control scores, with any response time. (n 192).
Figure 10: Zero-shot accuracy data human annotators with perfect NLI Control scores, with any response time. (n = 409).

---

Additional Figures and Data for Zero-Shot Experiment
G.1 Accuracies Per Instruction
0.54
0.5
0.78
ILAS
I-27
0.00
TU:70
01:84
0.25
0.23
DU40
ABSA
0,00
0.52
WABS
0L25
W.P5
10,00
L.00
10,001
0L75
OL,7
TURO
70++
0.00
11.00
3.00
F1 AAA-N
60.00
DUB00
0.5
0.50
QU422
[X00
10775
0.5
BU:0
0.60
8.60
10:00
0.00
1.00
1100
0.97
0.83
1.03
0.53
0.75
TUR7
CPP-
0,40
0.40
0.20
0.00
0.00
0.00
0.00
H.R,
dowr, Hiss, Ingly
geazanned Uner
Juaiifed Jo. satting
Aball... 3m
sprembalt
funmatte
fertiment
statt Jwiiit
paraphaage
fome, meroning
fiesch
grammatiiall
eathypporo
zoating
Inflection
butaashesid
cusccal notical Apun
Repecr
instuctive
misleading
invlevanc
aull
Figure 14: Zero-shot NLI accuracy of human annotators models, by specific instruction (see $H.1 for instruction templates). Bars are colored by instruction category.

---

Instructions and Examples
H.1
All Instructions
category
name
instruction
instructive
MNLI-YN
(sentence1) ] Using only the above description and what you know about the world, "[sentence2]" is defimitlly correct. Yes, no or unclear?
instructive
justified-in-sayinn is-it-true
[sentencel] } Are justified in saying that "(sentence2)
instructive
(sentencel) Based the previous passage, is it true that "(sen- tence2)
instructive
guaranteed- does-this-imply
[sentencel] ] Then, is it guaranteed true that " ((sentence2]
instructive
[sentencel] ] Question: Does this imply that " "(sentence2)""
misleading
words-appear
[sentencel] Do most of the words in the following sentence appear in the above paragraph? (sentence2]
misleading
sinilar-words
(sentencel) Are there lots of similar words between the above passage and the following sentence " "(sentence2)"
misleading
same-meaning
(sentence1) Does the above have the same meaning as "(sen- tence2]"?
misleading
paraphrase
(sentencel) } Can the above be paraphrased as: "(sentence2)" "?
misleading
summarize
[sentencel] } Can the above be summarized as "(sentence2)""
misleading
start-with-the
[sentencel] | Does the previous paragraph start with "the"? (sen- tence2]
misleading
grammatical
{sentencel] | Is the next sentence grammatically correct? (sen- tence2)
misleading misleading
sentimment sportsball
(sentencel) the above paragraph positive review? (sentence2)
(sentencel) ] Is the above paragraph piece of sports news? (sen- tence2)
misleading
french
(sentencel) } Is the above text French? (sentence2]
irrelevant
zoning
[sentence|| Single-family zoning is bad for American cities. "[ "ssetennc2)]"
|sentencel| } Inflections are annoying and thank god that Middle English got rid of most of them. "(sentence2)"?
irrelevant
inflection
irrelevant
gauss
[sentence]] When Bolyai sent Gauss his discovery of non- Euclidean geometry, Gauss replied that arrived at the same results 30 years ago. "(sentence2)"?
(sentence1) ] If bonito flakes boil more than few seconds, the stock becomes too strong? "(sentence2)
irrelevant
katsuoboshi
irrelevant
euthyphro
(sentence the pious loved by the gods because it is pious? Or pious because loved by the gods? " "(sentence2)")
null null
concat-phm
[sentencel] } {sentence2 | scntencel]
[sentence2] }
concat-hpm
Table 3: All prompts used in the main text of the paper. All templates use "Yes"/"No" as target words the entailment and non-entailment classes, respectively.

---

H.2
Example Sources for Test Condition
category
dataset
remarks
instructive
MNLI-YN
RTE. MNLI
6 entailment labels, 4 contradiction labels, 2 neutral labels were chosen; the latter two map to "Na" answers.
instructive
mustifiedin--ayingg
instructive
is-it-true
instructive
guaranteed-true
instructive
does-this-imply
mislcading
words-appear
RTF
Examples were handpicked such that misleading task would have a "Yes" label while NLI task had "No" label. From RTE labels, contradiction labels and 2 neutral labels were chosen and mapped to the "No" labels.
misleading
similar-words
misleading
same-meaning paraphrase summarize
RTE
Examples were handpicked such that the misleading task would have a "No" label while NLI task had "Yes" label- i.e., through examples where the second sentence was indeed an entailment but only tangential to the main point of the first sentence.
misleading misleading
misleading
start-with-the
RTE
All examples were such that the premise paragraph did indeed start with 'the' but the hypothesis sentence not entailed, so misleading task answer was "Yes" while the NLI task answer was "No". 3 contradiction labels and neutral labels were chosen map "No" labels.
misleading
grammatical
RTF
Grammatically correct but non-entailing examples from RTE were chosen such that the misleading task answer is "Yes" while the NLI task answer is "No". contradiction labels and 2 neutral labels chosen to map "No" labels.
misleading sentiment
Amazon Polarity (Zhang al., 2015)
Reviews were taken from the Amazon Polarity dataset as premise paragraphs. hypothesis sentence was manually written based on the review. If the review was positive, the hypothesis sentence was entailed; if the review negative the hypothesis sentence was entailed. There were 3 non-cntuilments and 2 entailments.
misleading
sportsball
RTF
RTE examples that had nothing related sports were chosen, such that the misleading task answer is "No" while the NLI task answer "Yes".
irrelevant
zoning
RTE, MNLI
6 entailment labels, 4 contradiction labels, 2 neutral labels were chosen; the latter two map "Na" answers.
irrelevant
inflection gauss
irrelevant
irrelevant
katsuoboshi
irrelevant
euthyphro
null
concat-phm
RTF, MNLI
entailment lahels, contradiction labels, 2 neutral labels were chosen; the latter two map to "Na" answers.
null
concat-hpm
Table 4: All source datasets used for each prompt for the main text of the paper. All templates use "Yes"/"No" as target words for the entailment and non-entailment classes, respectively. For RTE examples, we collapse the SuperGLUE version's "neutral" and "contradiction" to "non-entailment" such that all of our tasks are binary classification. We balance the distribution of "contradiction" and "neutral" labels within our study's 's non-entailed items.

---

H.3
Example Sources for General Controls
category
name
dataset
remarks
misleading
start-with-the
RTE
examples were that the premise paragruph did indeed start with 'the' but the hypothesis sentence was not entailed, so the misleading task unswer was "Yes" while the NLI task unswer was "No". 5 contradiction lahels were chosen to map to "No" labels.
misleading
grammatical
BLiMP (Warstadt et al., 2020)
Grammatically incorrec sentences from BLiMP were used as hypothesis sentences, while grammatically correct premise para- graphs were handwritten for the sentence, such that the misleading "Yes".
task
"No" while NI.I task
misleading sentiment
Yelp Polarity (Zhang et al., 2015)
Reviews were taken from the Yelp Polarity dataset as premise paragraphs and hypothesis
manually bused the review. If the review was positive, hypothesis sentence not entailed: the review was negative hypothesis sentence was entailed. There were 3 non-entailments and entailments.
misleading sportsbull
HuffPost
(Misra
articles in 'Spurts' category the dataset and non-entailing hypothesis sentences were manually written, such that the mislcading task answer was "Yes" while the NLI task answer was "No".
Excerpts were
Grover, 2021; Misra,
2022)
misleading french
XNLI (Conneau et al., 201 8)
Non-entailing French XNLI examples were taken such that the misleading task answer was "Yes" while the NIJI task answer was "No".

---

I
Results on Zero-Shot Experiment Control Conditions
I.1
NLI Controls
0.04
0.03
0.04
0.03
1
0.16
0.18
0.22
0.27
0.22
0.84
0.8
0.78
0.75
0.75
Percentage of Subjects
0.69
0.6
0.4
0.2
mnli yn (103)
is_it. true (97)
does. this imply (100)
guaranteed, true (107)
justified in, in_saying (97)
Score Attained
Figure 15: Subjects' scores the NLII control condition (n 504, only subjects whose completion times above floor cutoff) Each bar represents the breakdown of percentage of subjects assigned the prompt who scored 0.
and 2 out two NLI control items presented.

---

1.2
General (Surface Task) Controls
0.43
0.47
0.54
0.12
0.13
0.08
0.08
0.30
0.29
0.27
0.28
0.35
0.44
0.23
0.8
Percentage of Participants
0.18
0.64
0.65
0.67
0.64
0.6
0.33
0.27
0.54
0.52
0.35
0.44
0.4
0.27
0.24
0.2
0.12
0
paraphrase (49)
summarize (49)
same. meaning (52)
similar, words (52)
words. appear (53)
french (49)
sentiment (52)
sportsball (48)
grammatical (50)
start. with the (50)
Score Attained
Figure 16: Subjects' scores on the general suface task controls (n = 504, only subjects whose completion times were above floor cutoff) Each bar represents the breakdown of percentage of subjects assigned the prompt who scored 0, 1 and 2 out of two general control items presented; subjects were scored on correctly performing the misleading task.

---

J Effect of Experience with Prior NLP Studies
005
We conducted pilot (n - 29) to assess the effect of prior exposure to NLP studies on humans' inter- pretation of prompt instructions. In this pilot, select only subjects with prior experience with NLP studies. All participants had to first screen through pre-test with the question "How many mTTurk tasks have you completed for language re- search (e.g. Stanford NLP Group, MIT NLP Group, NYU NLP Group, etc.)?". Only participants who selected the option "None" were qualified to con- tinue to take the study. 66 participants took the pre-test and 29 qualified subjects. We compare these results
Accuncy
0.75
Janted Mi sayoo
gereiend toe
Hils leply
Figure 17: NLJ Control scores of subjects (1 29) with no prior NLP experience.
earlier pilot (n 67) that used the same prompts and examples, with no filtering participants based on previous exposure.
0.1
1.15
Accunsty
0.75
0.5
0.61
Comparing control condition scores (Figure 17 vs. Figure 18 for NLI Controls' Figure 19 vs. Fig- ure 20 for General Controls'), subjects without prior exposure score higher on both the mislead- ing task and NLI task (recall that in the controls, subjects scored performing the surface task explicitly described by the prompt). Comparing test condition scores (Figure 21 vs. Figure 22), sub- jects without prior exposure perform better at the NLI task when instructions are instructive and dra- matically worse when instructions are mislcading, compared to the sample that was not controlled for exposure. These results suggest that subjects with- out prior exposure appear to follow explicit task instructions more closely than the sample that was not controlled for exposure.
K.R. Sto
Janthed St saying
Baseeord how
fiis Soply
Figure 18: NLI Control scores of sample of subjects (n = 67) that were not filtered on prior exposure to NLP tasks.
1030
1.00
1.30
1.50
Accuncy
0.40
KLSO
The behavior of subjects without prior exposure to NLP studies is similar to the result when we select subjects that score perfectly on the General Controls (Figure 6) 6)-suggesingg that specifying for NLP-study inexperience may select for sam- ple of humans who follow task instructions more strictly. While we leave full study that controls for exposure prior NLP studies to future work, we predict that it will only strengthen the trend for misleading prompts seen our main results (namely, that humans do poorly on the actual task if given misleading prompts).
Jvet
Brencential
Spumball
Frimd
wouitt uprter
Aatobat pearts
Josstos
Aerglbate
Figure 19: General Control scores of subjects (n 29) with no prior NLP experience.
10.27
10.25)
RLS7
Accurnyy
ILCA
n3
KL22
20
Dset. wilk
Prosttattall
Woitaret
Summaal
freec
Words uppee
Parploan
Figure 20: General Control scores of sample of subjects (n 67) that were not filtered on prior exposure NLP tasks.

---

1.00
0.6
0.50
0.2
10. true
jusnifiedd In. sayiog
guaranteed, tree
mali daes, this, Imply
start with the
grammatical
sentitent
sparsball
french
words appear
similar, words
parophrase
simamarizt
inflection
zuttin
euthyphhn
battonhhshi
nulj
meaning
Instrettie
misleading
inrelevant
mull
Figure 21: Per-instruction accuracy on the test condition item of subjects with no prior NLP experience (n 29).
1.00
1.00
1.00
1.00
TU.00
1.00
0.8
0.80
0.75
0.755
Accuracy
0.67
0.50
0.50
0.4
0.33
0.33
0.2
it. tree
Jusufiee in. seying
Buaaramedd, true
nual dres,, this Imply
start, with, the
Brammetical
sentimant
sponshall
freach
words, appear
similla words
same meennin
parophirase
summarise
Inflectio
zoning
eathyphro
katsookoshi
Eauss
null
instructive
misleading
inrelevant
null
Figure 22: Per-instruction accuracy on the test condition item of subjects that were not filtered based on prior exposure to NLP tasks 67).

---

K
Few-Shot Experiment Post-Experiment Survey
S/N
Prompt Category
How did you decide choose "Yes" or "No"?
What did you think about the instructions gave?
Few-Shot With Labels
Mis-
In the first few questions, my strategy is to read through the entire paragraph or sentence and then decide whether the paraphrased sen- tence makes sense or not. However, then I started to look at the paraphrased sentence first and decide whether it correct wrong based on the given piece of text. Initially, I also considered whether the paraphrased sen- tence captured all the major details not, but the quiz later shows that comprehensiveness not factor.
I'd say the instructions are not quite direct? In my opinion, it would make more send to ask if the given sentence is correct or not than ask if it paraphrases the text.
Moderate
Mis- Extreme
chose my answer based on what believed was correct.
don't really like the question, "Is this gram- matically correct". Some were definitely not grammatically correct (capitalization errors, past/present tense), but the answer was still yes. feel like the question should be changed because it seems like the question is actually, "Is this statement true based on the context given in the paragraph".
Few-Shot Without Labels
Irrelevant
tried to see whether what was stated in the question was consistent with the preceding sentences. Sometimes it involved logical de- duction, and other times it was not implied all by the other sentences but just related. Sometimes was unsure what to choose be- cause the premise of the question was wrong.
was confused because that statement was included every question, but didn't seem relevant.
Mis- Moderate
I'm looking for whether the information pro- vided in the first half can more or less en- capsulated by the second half, meaning that if one were to read the first half and another the second, they would come away same conclusion.
There a level of ambiguity at first as con- sidered what exactly entailed: whether or not its "correct" statement given the con- text is a confounding factor, when it shouldn't influence whether not its a good paraphras- ing.
Mis- Extreme
looked at whether the sentence was accurate to the information given in the text, and also the sentence itself had correct grammatical structure. It was little difficult because of the sentences made inferences that weren't i explicit given text, so wasn't 't sure that was grammatical error not.
Usually, T think of something being gram- matically correct when the sentence has cor- rect grammatical structure, including punc- tuation and capitalization. Since most of the sentences seemed to fit this, I thought that maybe grammar also encompasses ity of the statement based the text, chose my answers based on that.
valid-
6
Mis- Extreme
I chose "Yes" when the shorter sentences present accurate information from the longer sentences. was kind of confused about the question because most of the sentence (maybe all) seemed to be grammatically correct.
For the first two questions, T was paying at- tention to whether the sentences were actually grammatically correct. Later on, just tried to see if the shorter sentences give accurate information based on the longer parag traphs above.
Table 5: Sample of free-text responses of subjects to the questions "How did you decide to choose "Yes" or 'No"?" and "What did you think about the instructions we gave?". Responses were elicited from subjects after they answered 32 items with their assigned prompt. Each table row indicates responses from unique subject.


Reviews:
[{"review_id": "DVeJdwVK0r", "reviewer": "Reviewer_3rDv", "summary": "", "questions": "A. Why do we need LLMs or human annotators to fulfill the task with misleading or irrelevant prompts in a zero-shot manner if the task is not clearly described at all? More examples are needed to clarify the motivation and the experimental results of this paper. \n\nB. What does it mean if LLMs achieve higher accuracy than humans in NLI when the instruction actually guides them to do another task without any information about NLI? Isn't it natural for humans to be faithful to the instruction? Why is this phenomenon complex?\n", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the in-depth review of our paper. \n\n## Response to criticism #1: Concerns about the scope of the study\n\nWe agree that the scope of our study is limited, which is why we are submitting this as a short paper. Our goal with this paper is to call attention to an interesting empirical trend that runs counter to common intuitions, and thus motivate future work on similar types of questions. In particular, we are trying to highlight the need for future work to empirically vet human behavior rather than comparing models to idealized/assumed human behavior.\n\nThe scope is small because this is a highly controlled study that entailed significant human effort to design well. We highlight that all our examples are handpicked so that the answers for an NLI task and answers for the explicitly instructed non-NLI task—in the case of misleading instructions—are always separable. We describe reasons for why the choice of NLI as our task is because it is the in-principle best task for our hypothesis in our Limitations section, and acknowledge that there is certainly room for future work in comparing human v. model instruction-following behaviors signatures on other tasks. However, we believe the current results are pithy but sufficient to show that human behavior signatures may be more complex than can be intuitively predicted, and hence hope to motivate further work by submitting this as a short paper.\n\n## Response to criticism #2: Concerns about the claim of the paper\nBased on your question, **we believe you misunderstood the role of human subjects in our study and our overall research question:** We are trying to determine whether humans and models exhibit similar behaviors *when they are given misleading or irrelevant instructions.* This question is important to answer because there are multiple papers analyzing and criticizing models’ behavior in such a setting (see our Related Work section), but there is no data on what humans would do in comparable situations. We are *not* collecting the typical sense of a “human baseline” (which serves to judge if models perform better/worse than humans on a specific NLP task and in which case we agree informative instructions are important)\n\nTo answer our research question, we not only need to collect instruction-following behavior in the best case, where “it is fundamental to describe the task clearly first”, but also in the worst cases: when (i) there are bad descriptions of the task, which are our misleading prompts, and (ii) when there are no descriptions of the task, which are our irrelevant and null prompts.\n\nOur research question also motivates why our experiments are zero-shot. We are testing instruction-following behavior, and not the task-deduction behavior which may be engaged when given few-shot examples. That is, we are giving minimal auxiliary information so as to test the intrinsic biases humans and models have in interpreting instructions. \n\nYou claim that our conclusions are not insightful because it is “natural for humans to be faithful to the instruction”. **This is the kind of (intuitive) assumption that the paper is testing empirically, and in fact, finds does not always hold.** Namely, counter to this intuition, humans perform well with irrelevant instructions. The W&P paper which we are responding to also assumed this and predicted that humans would hence perform well if given instructive instructions, and badly if given misleading, irrelevant or no instructions. We show that that is not in fact what happens.\n\nOur results also show interesting subtleties in human v. model behaviors that relying on intuition alone would not reveal. For instance, while models show a strong NLI performance variance with misleading instructions based on how proximal the distractor task is to the NLI task (e.g. models are more likely to perform the NLI task when asked to do the summary task than when asked to do the grammaticality task), humans are more uniformly unlikely to perform the NLI task no matter the distractor task (compare the variance of Humans’ misleading-prompt scatter points and that of models in Figure 2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Large language models (LLMs) have shown impressive performance in various NLP tasks by following prompts. \nThis paper studies \"whether language models are worse than humans at following prompts\"  and has some findings different from previous works, which might be helpful for the LLM community to analyze the behavior of humans and LLMs.", "reasons_to_reject": "1. LLMs are versatile in general-purpose NLP tasks, whereas this paper only conducts experiments in the NLI task using less than 200 examples (Table 2). The experimental results fluctuate too much in Figure 2, especially for misleading prompts. The result analysis and discussion are not so informative and helpful for future work.\n\n2. The claim of this paper is a little weird. When humans are involved in evaluating NLP models, it is fundamental to describe the task clearly first. I don't understand the point of this paper, which claims \"human behaviors are complex (given misleading prompts)\" with few insightful conclusions.", "questions_to_authors": "A. Why do we need LLMs or human annotators to fulfill the task with misleading or irrelevant prompts in a zero-shot manner if the task is not clearly described at all? More examples are needed to clarify the motivation and the experimental results of this paper. \n\nB. What does it mean if LLMs achieve higher accuracy than humans in NLI when the instruction actually guides them to do another task without any information about NLI? Isn't it natural for humans to be faithful to the instruction? Why is this phenomenon complex?\n", "paper_topic_and_main_contributions": "This paper investigates the instruction-following ability of language models and humans by comparing their performance in the NLI task under different prompt settings: instructive, misleading, and irrelevant.\n\nContributions: This paper empirically finds that humans and models both perform well with irrelevant instructions. In addition, when given deliberately misleading instructions, humans follow the instructions faithfully, whereas models do not. \nThese findings caution that future research should not idealize human behaviors as a monolith and should not train or evaluate models to mimic assumptions about these behaviors without first validating humans’ behaviors empirically.\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "achma3S6F6", "reviewer": "Reviewer_fzxT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your kind comments on our paper!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting research question. \n- Comprehensive human studies to illustrate the point. \n", "reasons_to_reject": "NA", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper follows W&P's previous work on comparison of human and model behavior in terms of following different prompts (misleading, irrelevant, null). \n\nThe paper found that it is not necessary true in previous assumptions of human behavior through human studies. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Au86SofdMy", "reviewer": "Reviewer_YaYe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper! We find your feedback valuable in improving the clarity of our work. To address the your following comments:\n\nWe agree that the scope of our study is limited, which is why we are submitting this as a short paper. Our goal with this paper is to call attention to an interesting empirical trend that runs counter to common intuitions, and thus motivate future work on similar types of questions. In particular, we are trying to highlight the need for future work to empirically vet human behavior rather than comparing models to idealized/assumed human behavior.\n\nOn scope— we agree that it would be valuable to explore tasks other than NLI. The scope is small because this is a highly controlled study that entailed significant human effort to design well. For one, the small scope is so that we can hand-pick examples for a rigorous experimental design—namely, that our examples for misleading prompts are chosen such that the answers for the explicit task and NLI task are always separable. Furthermore, NLI is the only task that was systematically held out from training by the models we tested.\n\nWe certainly agree that extending the findings by validating other tasks would be a worthwhile pursuit. For this short paper submission, however, we chose to be focused and rigorous in favor of breadth, and hope that the results motivate future work in expanding to other tasks.\n\nThank you again for your review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting subject to investigate LLMs and compare them to humans on different instruction artifacts.\n2. Detailed experimentation and the result about A2 not being consistent with human behavior is interesting.\n3. Contributes human data for the experiments which can be used for future works if authors make it open-source.", "reasons_to_reject": "1. Results are interesting but limited in scope and impact. I can’t see where the findings of this paper will be useful. \n2. Moreover, the only evaluated here is NLI. Evaluation on the other NLP tasks will help the paper to establish the generality of their findings.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates the belief that humans would struggle with abnormal instructions. Their results show that humans reliably ignore irrelevant cues and perform well in tasks. However, when faced with misleading instructions, humans follow faithfully, unlike models. This result is important in cautioning against assuming uniform human behavior and underscores the importance of validating human behavior before training models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2q1BrT2qDh", "reviewer": "Reviewer_k3cH", "summary": "", "questions": "A. Why do you collapse misleading-moderate and misleading-extreme?\nB. Why do you choose W&P's work to follow up?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review of our paper! We agree that the scope of our study is limited in that it critiques mainly one past paper. However, we believe the current results are pithy and sufficient to show that human behavior signatures may be more complex than can be intuitively predicted, and hence by submitting this as as a short paper we hope to motivate further work to test assumptions in other studies that make claims comparing humans to models on the basis of intuited rather than observed human behavior.\n\nWe collapse misleading-moderate and misleading-extreme because the overall patterns from both were the same *with respect to this paper’s hypothesis*—that models perform too well on the NLI task relative to humans when given misleading prompts. For the sake of argumentative clarity and conciseness, we decided to collapse the discussion of the two categories.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Conduct empirical experiment to show that one assumption (in a previous paper) about human performance is wrong. This is a good finding because it underscores the importance of having more rigorous study before making these assumptions if one were to evaluate models in terms of “human-likeness”.\n2. Sound experimental design.", "reasons_to_reject": "1. A lot of work is built on W&P’s work, this paper mainly evaluates on human performance and provide some analysis.\n2. Only follows up on W&P's work. In the introduction section, this paper identified several papers that made similar observations (i.e. models perform well when given non-ideal input). It would be nice to see how often the assumptions are violated. ", "questions_to_authors": "A. Why do you collapse misleading-moderate and misleading-extreme?\nB. Why do you choose W&P's work to follow up?", "paper_topic_and_main_contributions": "This paper conducts a follow-up study on the work by Webson and Pavlick (2022). W&P found that when presented with misleading prompts, LLMs perform just as well as if they were given good prompts. W&P further question whether models understand task instructions the same way as humans do. In this paper, the authors measured human performance on datasets (constructed with the same strategy that W&P used) and found that one of W&P’s assumptions do not hold for human performance. They caution future research to validate assumptions on human performance, if the research compares models with humans. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Pros:
- It is valuable to assess the implicit assumptions in prompt perturbation about the correct or human way to handle a misleading or irrelevant prompt. They find the "human" way to handle things is not to execute misleading instructions, but to make reasonable adjustments,  and to ignore irrelevant information. The authors argue that papers that test the consistency of prompting make this assumption often, but it has never been tested in humans.
- This paper is a direct response to a specific work on NLI, and mimics their setup, which justifies the focus on a single task.

Cons:
- Results are limited to only NLI. Not only does this make the title overly broad, but as 3rDv pointed out, the particular nature of the NLI task could easily shape the findings because it's not clear how to handle certain perturbations without direct instructions.
- Real world annotation is often performed with complex instructions, so it is hard to see whether these results would generalize in more complex settings.
Decision: Accept-Findings