Reference	Task	Questions and guidelines
Camburu et al. (2018)	Natural language inference	We encouraged the annotators to focus on the non-obvious elements that induce the given relation, and not on the parts of the premise that are repeated identically in the hypothesis. For entailment, we required justifications of all the parts of the hypothesis that do not appear in the premise. For neutral and contradictory pairs, while we encouraged stating all the elements that contribute to the relation, we consider an explanation correct, if at least one element is stated. Finally, we asked the annotators to provide self-contained explanations, as opposed to sentences that would make sense only after reading the premise and hypothesis.We did in-browser checks to ensure that each explanation contained at least three tokens and that it was not a copy of the premise or hypothesis. We further guided the annotators to provide adequate answers by asking them to proceed in two steps. First, we require them to highlight words from the premise and/or hypothesis that they consider essential for the given relation. Secondly, annotators had to formulate the explanation using the words that they highlighted. However, using exact spelling might push annotators to formulate grammatically incorrect sentences, therefore we only required half of the highlighted words to be used with the same spelling. For entailment pairs, we required at least one word in the premise to be highlighted.For contradiction pairs, we required highlighting at least one word in both the premise and the hypothesis. For neutral pairs, we only allowed highlighting words in the hypothesis, in order to strongly emphasize the asymmetry in this relation and to prevent workers from confusing the premise with the hypothesis. We believe these label-specific constraints helped in putting the annotator into the correct mindset, and additionally gave us a means to filter incorrect explanations. Finally, we also checked that the annotators used other words that were not highlighted, as we believe a correct explanation would need to articulate a link between the keywords.
Do et al. (2020)	visual NLI	similar to Camburu et al. (2018)
Kim et al. (2018)	self-driving cars	We provide a driving video and ask a human annotator in Amazon Mechanical Turk to imagine herself being a driving instructor. Note that we specifically select human annotators who are familiar with US driving rules. The annotator has to describe what the driver is doing (especially when the behavior changes) and why, from a point of view of a driving instructor. Each described action has to be accompanied with a start and end time-stamp. The annotator may stop the video, forward and backward through it while searching for the activities that are interesting and justifiable.
Zhang et al. (2020)	coreference resolution	Given a context and a pronoun reference relationship, write how you would decide the selected candidate is more likely to be referred than the other candidate using natural language. Don’t try to be overly formal, simply write what you think. In the first phase, we ask annotators to provide reasons for all WSC questions. Detailed instructions are provided such that annotators can fully understand the task1. As each question may have multiple plausible reasons, for each question, we invite five annotators to provide reasons based on their own judgments. A screenshot of the survey is shown in Figure 3. As a result, we collect 1,365 reasons. As the quality of some given reasons might not be satisfying, we introduce the second round annotation to evaluate the quality of collected reasons. In the second phase, for each reason, we invite five annotators to verify whether they think the reason is reasonable or not2. If at least four annotators think the reason is plausible, we will accept that reason. As a result, we identify 992 valid reasons.
Lei et al. (2020)	future event prediction	we also require them to provide a rationale as to why it is more or less likely
Da et al. (2020)	harm of manipulated images	For each question we require annotators to provide both an answer to the question and a rationale (e.g. the physical change in the image edit that alludes to their answer). This is critical, as the rationales prevent models from guessing a response such as “would be harmful” without providing the proper reasoning for their response. We ask annotators to explicitly separate the rationale from the response by using the word “because” or “since” (however, we find that the vast majority of annotators naturally do this, without being explicitly prompted).
Ehsan et al. (2019)	gaming	“Please explain your action”. During this time, the player’s microphone automatically turns on and the player is asked to explain their most recent action while a speech-to-text library automatically transcribes the explanation real-time.