SlowGuess/NAACL_2022_OCR huggingface dataset:
Reference	Task	Questions and guidelines
Jansen et al. (2018)	science QA	Specific interfaces were designed. For a given question, annotators identified the central concept the question was testing, as well as the inference required to correctly answer the question, then began progressively constructing the explanation graph. Sentences in the graph were added by querying the tablestore based on key- words, which retrieved both single sentences/table rows, as well as entire explanations that had been previously annotated. If any knowledge required to build an explanation did not exist in the table store, this was added to an appropriate table, then added to the explanation.
Xie et al. (2020)	science QA	similar to Jansen et al. (2018)
Khot et al. (2020)	question answering	an-	The HIT here is to write a test question that requires CHAINING two facts (a science fact and some other fact) to be combined.
Jhamtani and Clark (2020)	question answering	an-	We then use (Amazon Turk) crowdworkers to annotate each chain. Workers were shown the question, correct answer, and reasoning chain. They were then asked if fact 1 and fact 2 together were a reasonable chain of reasoning for the answer, and to promote thought were offered several categories of “no” answer: fact 1 alone, or fact 2 alone, or either alone, justified the answer; or the answer was not justified; or the question/answer did not make sense. (Detailed instructions in the appendix)
Inoue et al. (2020)	question answering	an-	1. Read a given question and related articles. 2. Answer to the question solely based on the information from each article. 3. Describe your reasoning on how to reach the answer. Each reasoning step needs to be in a simple subject-verb-object form (see example below). Your reasoning must include sentences containing your answer.
Table 10: Questions that prior work uses to solicit human explanations in proximal mechanisms (in structured explanations).
Reference	Task	Questions and guidelines
Srivastava et al. (2017)	concept learning	The screenshot includes both “explanations” and “instr- tions”, however, we cannot infer the exact question
Hancock et al. (2018)	relation extrac- tion	we cannot infer the exact question
Table 11: Questions that prior work uses to solicit human explanations for procedure (in free text).
Reference	Task	Questions and guidelines
Lamm et al. (2020)	question answering	an-	referential equality, we cannot infer the exact question
Ye et al. (2020)	question answering	an-	Please read carefully to get accepted! (1) You're not required to answer the question. The answer is already provided and marked in red. Read examples below carefully to learn about what we want! (2) Identify important short phrases that appear both in the question and in the context. Important: The two appearances of the phrase should be exactly the same (trivial differences like plural form or past tense are still acceptable). Important: Write sentences like Y is "Switzerland". Make sure there is no typo in what you quote. (3) Explain how you locate the answer with the phrases you marked; Only use the suggested expressions in the table in the bottom.
Geva et al. (2021)	question answering	an-	1) Creative question writing: Given a term (e.g., silk), a description of the term, and an expected answer (yes or no), the task is to write a strategy question about the term with the expected answer, and the facts required to answer the question. 2) Strategy question decomposition: Given a strategy question, a yes/no answer, and a set of facts, the task is to write the steps needed to answer the question. 3) Evidence matching: Given a question and its de- composition (a list of single-step questions), the task is to find evidence paragraphs on Wikipedia for each retrieval step. Operation steps that do not require retrieval are marked as operation.
Table 12: Questions that prior work uses to solicit human explanations for procedure (in structured explanations).