SlowGuess/NAACL_2022_OCR huggingface dataset:
Madhusudhan Aithal and Chenhao Tan. 2021. On positivity bias in negative reviews. In Proceedings of ACL.
Tariq Alhindi, Savvas Petridis, and Smaranda Muresan. 2018. Where is your evidence: Improving fact-checking by justification modeling. In Proceedings of the First Workshop on Fact Extraction and Verification (FEVER), pages 85-90, Brussels, Belgium. Association for Computational Linguistics.
Faeze Brahman, Vered Shwartz, Rachel Rudinger, and Yejin Choi. 2020. Learning to rationalize for nonmonotonic reasoning with distant supervision. arXiv preprint arXiv:2012.08012.
David A Broniatowski et al. 2021. Psychological foundations of explainability and interpretability in artificial intelligence.
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natural language inference with natural language explanations. In Proceedings of NeurIPS.
Samuel Carton, Qiaozhu Mei, and Paul Resnick. 2018. Extractive adversarial networks: High-recall explanations for identifying personal attacks in social media posts. In Proceedings of EMNLP.
Samuel Carton, Anirudh Rathore, and Chenhao Tan. 2020. Evaluating and characterizing human rationales. In Proceedings of EMNLP.
Jeff Da, Maxwell Forbes, Rowan Zellers, Anthony Zheng, Jena D Hwang, Antoine Bosselut, and Yejin Choi. 2020. Edited media understanding: Reasoning about implications of manipulated images. arXiv preprint arXiv:2012.04726.
Cristian Danescu-Niculescu-Mizil, Justin Cheng, Jon Kleinberg, and Lillian Lee. 2012. You had me at hello: How phrasing affects memorability. In Proceedings of the ACL.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. 2020. Eraser: A benchmark to evaluate rationalized nlp models. In Proceedings of ACL.
Virginie Do, Oana-Maria Camburu, Zeynep Akata, and Thomas Lukasiewicz. 2020. e-snli-ve-2.0: Corrected visual-textual entailment with natural language explanations. In *IEEE CVPR Workshop on Fair, Data Efficient and Trusted Computer Vision*.
Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O Riedl. 2019. Automated rationale generation: a technique for explainable ai and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 263-274.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. arXiv preprint arXiv:2101.02235.
Dan Goldwasser and Dan Roth. 2014. Learning from natural instructions. *Machine learning*, 94(2):205-232.
Anthony G Greenwald, Debbie E McGhee, and Jordan LK Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology, 74(6):1464.
Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, and Christopher Ré. 2018. Training classifiers with natural language explanations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1884-1895, Melbourne, Australia. Association for Computational Linguistics.
Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. 2019. A richly annotated corpus for different tasks in automated fact-checking. arXiv preprint arXiv:1911.01214.
Carl G Hempel and Paul Oppenheim. 1948. Studies in the logic of explanation. Philosophy of science, 15(2):135-175.
Naoya Inoue, Pontus Stenetorp, and Kentaro Inui. 2020. R4C: A benchmark for evaluating RC systems to get the right answer for the right reason. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6740-6750, Online. Association for Computational Linguistics.
Peter Jansen, Niranjan Balasubramanian, Mihai Surdeanu, and Peter Clark. 2016. What's in an explanation? characterizing knowledge and inference requirements for elementary science exams. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2956-2965.
Peter Jansen, Elizabeth Wainwright, Steven Marmorstein, and Clayton Morrison. 2018. WorldTree: A corpus of explanation graphs for elementary science questions supporting multi-hop inference. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Harsh Jhamtani and Peter Clark. 2020. Learning to explain: Datasets and models for identifying valid reasoning chains in multihop question-answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 137–150, Online. Association for Computational Linguistics.
Frank C Keil. 2006. Explanation and understanding. Annu. Rev. Psychol., 57:227-254.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of NAACL.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082-8090.
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. 2018. Textual explanations for self-driving vehicles. In Proceedings of the European conference on computer vision (ECCV), pages 563-578.
Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In EMNLP.
Mucahid Kutlu, Tyler McDonnell, Matthew Lease, and Tamer Elsayed. 2020. Annotator rationales for labeling tasks in crowdsourcing. Journal of Artificial Intelligence Research, 69:143-189.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466.
Vivian Lai, Han Liu, and Chenhao Tan. 2020. "why is 'chicago' deceptive?" towards building model-driven tutorials for humans. In Proceedings of CHI.
Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2020. Qed: A framework and dataset for explanations in question answering. arXiv preprint arXiv:2009.06354.
Eric Lehman, Jay DeYoung, Regina Barzilay, and Byron C Wallace. 2019. Inferring which medical treatments work from reports of clinical trials. In *NAACL*.
Jie Lei, Licheng Yu, Tamara Berg, and Mohit Bansal. 2020. What is more likely to happen next? video-and-language future event prediction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8769-8784, Online. Association for Computational Linguistics.
Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. 2018. VQA-E: Explaining, elaborating, and enhancing your answers for visual questions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 552-567.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunson. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146.
Tania Lombrozo. 2006. The structure and function of explanations. Trends in cognitive sciences, 10(10):464-470.
Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. 2013. Learning to parse natural language commands to a robot control system. In Experimental robotics, pages 403-415. Springer.
Richard E Nisbett and Timothy D Wilson. 1977. Telling more than we can know: verbal reports on mental processes. Psychological review, 84(3):231.
Michael I Norton, Samuel R Sommers, Joseph A Vandello, and John M Darley. 2006. Mixed motives and racial bias: The impact of legitimate and illegitimate criteria on decision making. Psychology, Public Policy, and Law, 12(1):36.
Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. 2018. Multimodal explanations: Justifying decisions and pointing to the evidence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8779-8788.
Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning.
Nazneen Fatema Rajani, Rui Zhang, Yi Chern Tan, Stephan Zheng, Jeremy Weiss, Aadit Vyas, Abhijit Gupta, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020. ESPRIT: Explaining solutions to physical reasoning tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7906-7917, Online. Association for Computational Linguistics.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In ACL.
Cansu Sen, Thomas Hartvigsen, Biao Yin, Xiangnan Kong, and Elke Rundensteiner. 2020. Human attention maps for text classification: Do humans and neural networks focus on the same words? In Proceedings of ACL, pages 4596-4608.
Shashank Srivastava, Igor Labutov, and Tom Mitchell. 2017. Joint concept learning and semantic parsing from natural language explanations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1527-1536, Copenhagen, Denmark. Association for Computational Linguistics.
Chenhao Tan, Lillian Lee, and Bo Pang. 2014. The effect of wording on message propagation: Topic-and author-controlled natural experiments on twitter. In Proceedings of ACL.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERIFICATION. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: verifying scientific claims. arXiv preprint arXiv:2004.14974.
Ziqi Wang, Yujia Qin, Wenxuan Zhou, Jun Yan, Qinyuan Ye, Leonardo Neves, Zhiyuan Liu, and Xiang Ren. 2020. Learning from explanations with neural execution tree.
Sarah Wiegrefe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable nlp.
Robert A Wilson and Frank Keil. 1998. The shadows and shallows of explanation. *Minds and machines*, 8(1):137-159.
Zhengnan Xie, Sebastian Thiem, Jaycie Martin, Elizabeth Wainwright, Steven Marmorstein, and Peter Jansen. 2020. WorldTree v2: A corpus of science-domain structured explanations and inference patterns supporting multi-hop inference. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5456-5473, Marseille, France. European Language Resources Association.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotq: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.
Qinyuan Ye, Xiao Huang, Elizabeth Boschee, and Xi-ang Ren. 2020. Teaching machine comprehension with compositional explanations. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1599–1615, Online. Association for Computational Linguistics.
Omar Zaidan, Jason Eisner, and Christine Piatko. 2007. Using "annotator rationales" to improve machine learning for text categorization. In Human language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of the main conference, pages 260-267.
Hongming Zhang, Xinran Zhao, and Yangqiu Song. 2020. Winowhy: A deep diagnosis of essential commonsense knowledge for answering winograd schema challenge. In ACL.
A Proxy Questions for Proximal Mechanisms and Procedure
Proximal mechanisms. In collecting proximal mechanisms, studies are more likely to ask explicitly versions of "why is [input] assigned [label]", compared to the case of evidence. However, they often need to provide structured guidelines. For example, Camburu et al. (2018) and Rajani et al. (2019) discussed the need to enforce word overlap as a way to improve the quality of human rationales. The specific requirements are quite different (see Table 6 and Table 7). There are also specific formulations of explanations, e.g., "What aspect/stereotype/characteristic of this group (often un-fairly assumed) is referenced or implied by this post?" in Sap et al. (2020). Finally, it is common that we cannot infer the exact questions asked (8/18 papers that collect explanations in free text).
Procedures. We cannot identify the exact questions in three of five papers for explicitly step-by-step procedures, which reflects the importance of reporting detailed annotation guidelines. As researchers collect step-by-step guidelines, Ye et al. (2020) and Geva et al. (2021) adopt very different decomposition for their problems (see Table 12).
B Detailed Proxy Questions
Table 4-12 show the instructions we find in prior work that detail the proxy questions. Camburu et al. (2018) and Rajani et al. (2019) collect both evidence and proximal mechanism. We include them in the tables for proximal mechanisms. Also, for question answering tasks, the difference between procedure and proximal mechanism can be subtle. We consider the collected explanations procedure if they aim to explicitly provide step-by-step guides directly grounded in the input.
Reference	Task	Questions and guidelines
Zaidan et al. (2007)	sentiment analysis	Each review was intended to give either a positive or a negative overall recommendation. You will be asked to justify why a review is positive or negative. To justify why a review is positive, highlight the most important words and phrases that would tell someone to see the movie. To justify why a review is negative, highlight words and phrases that would tell someone not to see the movie. These words and phrases are called rationales. You can highlight the rationales as you notice them, which should result in several rationales per review. Do your best to mark enough rationales to provide convincing support for the class of interest.
You do not need to go out of your way to mark everything. You are probably doing too much work if you find yourself go-ing back to a paragraph to look for even more rationales in it. Furthermore, it is perfectly acceptable to skim through sections that you feel would not contain many rationales, such as a re-viewer's plot summary, even if that might cause you to miss a rationale here and there.
Sen et al. (2020)	sentiment analysis	1. Read the review and decide the sentiment of this review (positive or negative). Mark your selection.
2. Highlight ALL words that reflect this sentiment. Click on a word to highlight it. Click again to undo.
3. If multiple words reflect this sentiment, please highlight them all.
Carton et al. (2018)	Personal attack detection	40 undergraduate students used Brat (Stenetorp et al., 2012) to highlight sections of comments that they considered to constitute personal attacks.
Lehman et al. (2019)	Question answering	Prompt Generation: Question answering & Prompt creators were instructed to identify a snippet, in a given full-text article, that reports a relationship between an intervention, comparator, and outcome. Generators were also asked to provide answers and accompanying rationales to the prompts that they provided; such supporting evidence is important for this task and domain.
The annotator was also asked to mark a snippet of text supporting their response. Annotators also had the option to mark prompts as invalid, e.g., if the prompt did not seem answerable on the basis of the article.
Thorne et al. (2018)	fact verification (QA)	If I was given only the selected sentences, do I have strong reason to believe the claim is true (supported) or stronger reason to believe the claim is false (refuted). If I'm not certain, what additional information (dictionary) do I have to add to reach this conclusion.
In the annotation interface, all sentences from the introductory section of the page for the main entity of the claim and of every linked entity in those sentences were provided as a default source of evidence (left-hand side in Fig. 2). We did not set a hard time limit for the task, but the annotators were advised not to spend more than 2-3 minutes per claim.
Table 4: Questions that prior work uses to solicit human explanations (evidence).
Reference	Task	Questions and guidelines
Khashabi et al. (2018)	Question answering	We show each paragraph to 5 turkers and ask them to write 3-5 questions such that: (1) the question is answerable from the pas- sage, and (2) only those questions are allowed whose answer cannot be determined from a single sentence. We clarify this point by providing example paragraphs and questions. In order to encourage turkers to write meaningful questions that fit our criteria, we additionally ask them for a correct answer and for the sentence indices required to answer the question.
Yang et al. (2018)	Question answering	Workers provide the supporting facts (cannot infer the exact question)
Hanselowski et al. (2019) Fact tion	verifica-Stance annotation. We asked crowd workers on Amazon Mechanical Turk to annotate whether an ETS (evidence text snippets) agrees with the claim, refutes it, or has no stance towards the claim. An ETS was only con- sidered to express a stance if it explicitly referred to the claim and either expressed support for it or refuted it. In all other cases, the ETS was consid- ered as having no stance. FGE annotation. We filtered out ETSs with no stance, as they do not contain supporting or refut- ing FGE. If an ETS was annotated as supporting the claim, the crowd workers selected only sup- porting sentences; if the ETS was anno- tated as refuting the claim, only refuting sentences were selected.
Kwiatkowski et al. (2019)Question answering	Long Answer Identification: For good ques- tions only, annotators select the earliest HTML bounding box containing enough information for a reader to completely infer the answer to the ques- tion. Bounding boxes can be paragraphs, tables, list items, or whole lists. Alterna- tively, annotators mark “no answer” if the page does not answer the question, or if the information is present but not contained in a single one of the allowed elements.
Wadden et al. (2020) Fact tion	verifica-An evidence set is a collection of sentences from the abstract that provide support or contradiction for the given claim. To decide whether a collection of sentences is an evidence set, ask yourself, “If I were shown only these sentences, could I reasonably conclude that the claim is true (or false)? 1) Evidence sets should be mini- mal. If you can remove a sentence from the evidence set and the remaining sentences are sufficient for sup- port / contradiction, you should remove it. 2) There may be multiple evidence sets in a given abstract. See more at https://scifact.s3-us-west-2.amazon.com/doc/evidence-annotation-instructions.pdf
Kutlu et al. (2020) relevance assessment	Please copy and paste text 2-3 sentences from the webpage which you believe support your decision. For instance, if you selected Highly Relevant, paste some text that you feel clearly satisfies the given query. If you selected Definitely not relevant, copy and paste some text that shows that the page has nothing to do with the query. If there is no text on the page or images led you to your decision, please type “The text did not help me with my decision”.
Table 5: Questions that prior work uses to solicit human explanations (evidence).
Reference	Task	Questions and guidelines
Jansen et al. (2016)	science QA	For each question, we create gold explanations that describe the inference needed to arrive at the correct answer. Our goal is to derive an explanation corpus that is grounded in grade-appropriate resources. Accordingly, we use two elementary study guides, a science dictionary for elementary students, and the Simple English Wiktionary as relevant cor-pora. For each question, we retrieve relevant sentences from these corpora and use them directly, or use small variations when necessary. If relevant sentences were not located, then these were constructed using simple, straightforward, and grade-level appropriate language. Approximately 18% of questions required specialized domain knowledge (e.g. spatial, mathematical, or other abstract forms) that did not easily lend itself to simple verbal description, which we removed from consideration. This resulted in a total of 363 gold expla-nations.
Rajani et al. (2019)	Question answering	Turkers are prompted with the following question: “Why is the predicted output the most appropriate answer?” An- notators were in- structured to highlight relevant words in the question that justifies the ground-truth answer choice and to provide a brief open-ended explanation based on the high- lighted justification could serve as the commonsense reason- ing behind the question. Annotators cannot move forward if they do not highlight any relevant words in the question or if the length of explanations is less than 4 words. We also check that the explanation is not a sub- string of the question or the answer choices without any other extra words. We collect these ex- planations from only one annotator per example, so we also perform some post-collection checks to catch examples that are not caught by our previ- ous filters. We filter out explanations that could be classified as a template. For example, expla-nations of the form “<answer> is the only option that is [correct-obvious]” are deleted and then reannotated.
Sap et al. (2020)	social bias	What aspect/stereotype/characteristic of this group (often un- fairly assumed) is referenced or implied by this post? — Use simple phrases and do not copy paste from the post.
Table 6: Questions that prior work uses to solicit human explanations for proximal mechanisms (in free text).
Table 7: Questions that prior work uses to solicit human explanations for proximal mechanisms (in free text).
Table 8: Questions that prior work uses to solicit human explanations for proximal mechanisms (in free text).
Reference	Task	Questions and guidelines
Ling et al. (2017)	algebraic problems	cannot infer the exact question
Alhindi et al. (2018)	fact verification	we cannot infer the exact question automatically extracting for each claim the justification that humans have provided in the fact-checking article associated with the claim. Most of the articles end with a summary that has a headline “our ruling” or “summing up”
Kotonya and Toni (2020)	fact verification	automatically scraped from the website, we cannot infer the exact question
Wang et al. (2020)	sentiment analysis & relation extraction	Turkers are prompted with a list of selected predicates (see Appendix) and several examples of NL explanations. We cannot infer the exact question
Brahman et al. (2020)	natural language inference	automatically generated. We cannot infer the exact question
Li et al. (2018)	visual QA	automatically generated, We cannot infer the exact question
Park et al. (2018)	visual QA	During data annotation, we ask the annotators to complete the sentence “I can tell the person is doing (action) because.” where the action is the ground truth activity label. However, We cannot infer the exact question in VQA-X.
Rajani et al. (2020)	physics reasoning	We cannot infer the exact question
Table 9: Questions that prior work uses to solicit human explanations for proximal mechanisms (in free text).