SlowGuess/NAACL_2022_OCR huggingface dataset:
Annotators must complete a qualifying exam in order to participate in the main annotation tasks. The qualifying exam consists of 3 HITs in the same format as the main absolute evaluation task for CommonsenseQA We pay $2.25 for the qualifying exam. There are 9-18 questions in total (3-6 questions per HIT), some of which are only answerable conditioned on previous answers. A user who answers "no" to question 3, for example, will not be asked to answer questions 4 and 5. Given the challenging and sometimes ambiguous nature of some of the questions, for the first run of the qualification exam, we manually awarded qualifications by inspecting the annotators' answers. Scores for the first run compared to our answers (out of 17 annotators attempting) ranged from 5 to 14 out of 18. The median accuracy was 11 out of 18, and we find that awarding the qualification to those with scores at or above the median aligns closely with our manual inspection. We thus use this score to assign qualifications in future iterations.
Because it is necessary that annotators understand the task before they can evaluate explanation quality (Wiegreffe and Marasovic, 2021), for tasks that are more difficult, i.e., NLI, we additionally require annotators to pass (score of 7 or above) a task-specific qualification exam with 8 questions, paid at $1.25.
In order to track quality throughout evaluation, we compute inter-annotator agreement using Krippendorff's  $\alpha$  and use a hidden built-in Javascript function to compute time per HIT spent. If any annotator completed the tasks in an unreasonably low time, or removing their annotations substantially improves Krippendorff's  $\alpha$ , we remove their annotations and re- annotate their instances. We additionally ensure that each experiment has a substantial number of distinct crowdworkers to mitigate individual annotator bias, reporting this as well as the mean and median number of HITs completed by each in Table 17.
B.6 Statistics
The number of distinct crowd annotators and the median and mean number of HITs completed for each experiment can be found in Table 17. More
the geographical requirements often still leads to  $>90\%$  of annotators in the US or Canada, due to AMT's pay structure being optimal in these countries.
Dataset	Split	#Instances by	Agreement	Total
0/3	1/3	2/3	3/3
Com.QA	Train	932	1078	1194	1296	4500
Dev	105	91	132	127	455
Test	298	227	328	397	1250
SNLI	Train	2372	805	621	702	4500
Dev	272	87	65	76	500
Test	678	225	166	181	1250
Test2	666	234	179	171	1250
Table 13: Statistics of our acceptability annotations.
detailed breakdowns of inter-annotator agreement for some experiments are in Tables 14 and 15.
C Absolute Evaluation by Label Accuracy
Can GPT-3 produce convincing explanations even for instances it cannot predict correctly? This has implications for model-generated explanations being "right for the right reasons". To produce label predictions, we follow the same prompt format as in Tables 2 and 11, removing the WHY? token and the gold explanations so that the model generates a label prediction instead. GPT-3 achieves  $50.8\%$  accuracy on CommonsenseQA compared to a  $20\%$  random baseline, and  $46\%$  accuracy on SNLI compared to a  $33.33\%$  random baseline.[18]
Figure 4 presents absolute evaluation results broken down by whether GPT-3 correctly predicts the instance label. The results show little variation between the correctly-predicted and incorrectly-predicted groups in most attributes. This indicates that GPT-3 explanations are not faithful enough to use in real-world applications in their current form.
D 2/3 Acceptability Statistics
When we treat explanations rated by at least  $2/3$  annotators as "acceptable", for CommonsenseQA,  $77.9\%$  of greedily-decoded explanations are acceptable; for SNLI,  $51.0\%$ .  $50.5\%$  of sampled explanations are acceptable; for SNLI,  $23.5\%$ . Out of the set of 5 (1 greedy + 4 stochastic),  $97.7\%$  of CommonsenseQA instances have at least one acceptable explanation, and  $79.5\%$  of SNLI.
Figure 4: Absolute evaluation results for explanations generated by GPT-3 based on whether GPT-3 predicted the instance label correctly or not. CommonsenseQA (top) and SNLI (bottom). See caption of Figure 2 and the Appendix-Table 19 for more details.
E Filter Model Details
We split the 4,955 distinct annotated explanations for CommonsenseQA (5000 for SNLI) into a train/dev set of 4500/455 (4500/500 for SNLI), where all 5 explanations for a given instance are placed in the same set to avoid leakage. We present statistics on the label distribution in Table 13. Along with the metric settings reported in the paper ("select-1" and explanation-level), we computed a metric that is instance-level but considers all explanations by computing metrics over the 5 explanations of an instance and then averaging across instances, finding in practice that the results are highly similar to the explanation-level evaluation.
We use Huggingface Datasets (Lhoest et al., 2021) and Huggingface Transformers (Wolf et al., 2020) for implementation. We format inputs to the models as follows, where expl is one of the five explanations and the gold_label is either 0 (not acceptable) or 1 (acceptable):
The T5-Large model is trained using a learning rate of  $1E - 4$  with linear decay, a batch size of 64, and default values for Adam (Kingma and Ba, 2015), gradient clipping, and dropout. We train for
Dataset	Split	Krippendorff's α
CommonsenseQA	Training + Validation	0.32
Test	0.40
SNLI	Training + Validation	0.51
Test	0.50
Test2	0.47
Table 14: Inter-annotator agreement for acceptability AMT studies.
a maximum 200 epochs, performing early stopping on the validation loss with a patience of 10 epochs. For T0-3B, we train with a batch size of 50. We use AdaFactor (Shazeer and Stern, 2018) with a linear warmup of 500 steps. We conduct an initial hyperparameter sweep over learning rate, considering  $1E - 5,5E - 05,5E - 06$ . The learning rate that achieves the best validation loss for the full-information model and the explanation-only model is  $1E - 5$ , which we use for all training experiments.
F Additional Filter Results
In the main experiments, at evaluation time, we labelled an explanation as acceptable if  $3/3$  annotators agreed on it. Here, we report results if this threshold is relaxed to  $2/3$ . Overall, the results are comparable: T0-3B outperforms the baselines according to both select-1 accuracy and AP (see Table 21 and Table 22).
<sup>18</sup>Low SNLI performance aligns with previous findings that GPT-3 struggles with sentence-comparison tasks (Brown et al., 2020; Zhao et al., 2021).
AMT Study	Dataset	Generality	Factuality	Grammar	New Info	Supports Label	Amount Info	Acceptability	Aggregate
GPT-3 Greedy	Com.QA	0.37	0.32	-0.01	0.09	0.45	0.21	0.28	0.38
GPT-3 Greedy	SNLI	0.25	0.57	0.39	-0.01	0.04	0.17	0.52	0.40
Dataset	CoS-E	0.71	0.38	0.36	0.42	0.68	0.08	0.25	0.59
Dataset	ECQA	0.01	0.21	0.30	0.00	0.03	0.25	0.04	0.20
Dataset	e-SNLI	0.37	0.23	0.37	-0.14	-0.12	0.04	0.15	0.19
GPT-3 Filtered	Com.QA	0.25	0.18	0.25	0.11	0.27	0.15	0.30	0.32
GPT-3 Filtered	SNLI	0.41	0.19	0.07	0.17	0.08	0.13	0.50	0.33
Table 15: Inter-annotator agreement for absolute-comparison AMT studies, using Krippendorff's  $\alpha$  computed on an interval scale from -1 to 1. The aggregate score is computed by treating the annotations along each attribute as separate instances and computing agreement across the entire set.
SNLI (Bowman et al., 2015)
Premise: There are two kilted men, one of them older and is holding bagpipes with the other one with a drum. Hypothesis: Two kiled (sp) men hold bagpipes Label: contradiction NLL-Predicted Explanation: The two kilted men are not holding bagpipes, they are holding a bagpipe and a drum.Filter-Predicted Explanation: Just because there are kilted men does not necessarily mean that they are holding bagpipes. This can be seen from the older kilted man is holding the bagpipes rather than the kilder (sp) one.
CommonsenseQA (Talmor et al., 2019)
Question: The hardcovers were especially tall, so he removed a shelf on the what to make room? Answer choices: hold alcohol, grocery store, bookcase, nightstand, or chest of drawers NLL-Predicted Explanation: Bookcases are tall, and so if they are full of hardcover books, they might need to be removed to make room for other things.Filter-Predicted Explanation: People keep books on bookshelves, and bookcases are very commonly used to hold books. Since hardcover books can be pretty tall, it takes up extra vertical space which would require removal of an upper shelf of the bookcase.
Table 16: Randomly-selected instances that our filter model predicts differently than NLL at the "select-1" task, and got incorrect but NLL got correct.
AMT Study	Task/Dataset	# Annotators	Median # HITs (Mean)
GPT-3 Greedy w/ Dataset Prompts vs. Dataset	Com.QA/CoS-E	16	31.5 (46.9)
Com.QA/ECQA	13	35 (57.7)
e-SNLI	12	39 (62.5)
GPT-3 Greedy: Author-written vs. Dataset Prompts	Com.QA/CoS-E	7	84 (107.1)
Com.QA/ECQA	13	49 (57.7)
e-SNLI	8	43.5 (93.8)
GPT-3 Greedy w/ Author-written Prompts vs. Dataset	Com.QA/CoS-E	8	90 (93.8)
Com.QA/ECQA	17	27 (44.1)
e-SNLI	8	93 (93.8)
GPT-3 Greedy (Absolute)	Com.QA	13	51 (57.7)
SNLI	12	14 (62.5)
Dataset (Absolute)	CoS-E	14	58 (53.6)
ECQA	19	7 (39.5)
e-SNLI	13	16 (57.7)
Acceptability (Training and Validation Data)	Com.QA (2973 HITs)	34	70 (87.4)
SNLI (3000 HITs)	14	128.5 (214.3)
Acceptability (Test Data)	Com.QA	17	32 (44.1)
SNLI	11	26 (68.1)
SNLI (Test2)	7	65 (107.1)
CoS-E	13	48 (57.7)
ECQA	16	38.5 (46.9)
e-SNLI	9	60 (83.3)
GPT-3 Filtered (Absolute)	Com.QA (468 HITs)	10	44.5 (46.8)
SNLI (273 HITs)	6	53 (45.5)
Table 17: Total # of annotators and mean # HITs completed per-annotator for each AMT study (out of 750 total # HITs unless otherwise specified = 3 annotators for each of 250 test instances).
Set of Test Explanations	Generality	Factuality	Grammar	New Info	Supp. Label	Amt. Info	Accept.
GPT-3 Greedy for Com.QA	0.90.4†	0.80.4(247)†	1.00.1‡	0.10.6	0.50.7(217)‡	-0.10.4(186)‡	0.50.6‡
CoS-E	-0.20.9	0.50.5(131)	-0.30.7	0.10.8	-0.30.9(190)	-0.50.5(78)	-0.90.4
GPT-3 Greedy for Com.QA	0.90.4√	0.80.4(247)‡	1.00.1‡	0.10.6	0.50.7(217)	-0.10.4(186)	0.50.6‡
ECQA	0.80.4	0.60.4(249)	0.10.7	0.60.5‡	0.70.5(247)^	0.50.5(239)‡	0.10.6
GPT-3 Greedy for SNLI	0.70.5^	0.70.5(246)	1.00.2†	0.10.6	-0.1*0.6	-0.20.4(203)	0.00.8
e-SNLI	0.60.6	0.80.4(236)	0.90.4	0.20.5√	0.20.5*‡	-0.10.4(238)^	0.70.4‡
Table 18: Statistics from the graphs plotted in Figure 2. Mean ± standard error presented; numbers in parenthesis indicate the number of datapoints, if not 250. *For SNLI, we modified the evaluation framework such that "Supports Label" was always answered instead of being conditioned on "New Info". Statistical significance results using a one-sided Wilcoxon signed-rank test at  $p$ -values of  $\ddagger = 0.00001$ ,  $\dagger = 0.0001$ ,  $\vee = 0.01$ ,  $\wedge = 0.1$  indicates that the median difference between the marked score distribution and the unmarked score distribution is greater than 0.
Set of Test Explanations	Generality	Factuality	Grammar	New Info	Supp. Label	Amt. Info	Accept.
Com.QA Pred. Correctly	0.90.3 (127)√	0.90.3 (126)√	1.00.1 (127)	0.00.6 (127)	0.70.6 (108)†	-0.10.4 (98)	0.50.6 (127)
Com.QA Pred. Incorrectly	0.80.4 (123)	0.70.5 (121)	1.00.1 (123)	0.30.6 (123)√	0.30.8 (109)	-0.10.4 (88)	0.50.7 (123)
SNLI Pred. Correctly	0.80.5 (115)∧	0.70.5 (112)	1.00.1 (115)∧	-0.10.7 (115)	-0.20.6 (115)	-0.30.4 (83)	-0.10.8 (115)
SNLI Pred. Incorrectly	0.70.5 (135)	0.70.5 (134)	1.00.2 (135)	0.20.4 (135)‡	0.10.5 (135)√	-0.20.4 (120)∧	0.10.8 (135)∧
Table 19: Statistics from the graphs of GPT-3 greedy explanations plotted in Figure 4. See the caption of Table 18 for further details.
Set of Test Explanations	Generality	Factuality	Grammar	New Info	Supp. Label	Amt. Info	Accept.
GPT-3 Greedy for Com.QA	0.90.4 (156)	0.80.4 (153)√	1.00.1 (156)‡	0.10.6 (156)	0.50.7 (135)	-0.10.5 (117)	0.30.7 (156)
GPT-3 Filtered for Com.QA	0.90.3 (156)∧	0.70.3 (155)	0.80.4 (156)	0.70.4 (156)‡	0.90.4 (154)‡	0.20.3 (152)‡	0.60.6 (156)√
GPT-3 Greedy for SNLI	0.80.4 (91)	0.60.6 (91)	0.90.3 (91)‡	0.00.6 (91)	-0.2*0.6 (91)	-0.20.5 (66)	-0.50.7 (91)
GPT-3 Filtered for SNLI	0.80.5 (91)	0.70.4 (88)	0.70.4 (91)	0.50.6 (91)‡	0.50.5* (91)‡	0.00.3 (89)†	0.10.8 (91)‡
Table 20: Statistics from the graphs plotted in Figure 3. See the caption of Table 18 for further details.
↓Model/Split→	“Select-1” Acc@2/3	Explanation-level AP@2/3
Dev	Test	Dev	Test
Random	57.30.4	57.90.4	56.20.5	58.00.9
Constant	—	—	56.9	58.0
NLL	79.1	79.6	77.5	75.0
T0-3B Expl.-only	77.13.5	75.81.2	75.62.0	77.31.4
T0-3B	86.60.9	85.80.7	85.60.5	87.00.8
Oracle Upper-Bound	97.8	97.6	100.0	100.0
Table 21: Results for acceptability classifiers trained on CommonsenseQA, with "acceptability" defined as: "at least 2/3 annotators labelled as acceptable." Subscripts indicate standard error over models trained with 5 different random seeds. The oracle upper bound is based on dataset properties (§3).
↓Model/Split→	“Select-1” Acc@2/3	Explanation-level AP@2/3
Dev	Test	Test2	Dev	Test	Test2
Random	28.20.5	27.80.2	28.00.1	28.10.9	27.60.3	28.30.6
Constant	—	—	—	28.2	27.8	28.0
NLL	51.0	51.2	50.4	47.7	47.5	46.1
T0-3B Expl.-only	47.01.0	50.52.1	50.62.8	48.91.4	45.21.5	44.92.1
T0-3B	57.81.9	60.31.5	59.22.3	66.73.3	64.73.3	67.13.6
Oracle Upper-Bound	76.0	81.2	77.6	100.0	100.0	100.0
Table 22: Results for acceptability classifiers trained on SNLI with "acceptability" defined as: "at least 2/3 annotators labelled as acceptable." Subscripts indicate standard error over models trained with 5 different random seeds. The oracle upper bound is based on dataset properties (§3).
You will read multiple-choice questions about daily life. For each question, an answer has been selected and 2 explanations for the answer have been given. Your task is to select the explanation that best explains the answer.
Each HIT contains 3 parts:
A question such as "If a lantern is not for sale, where is it likely to be?"
A selected answer, such as "house" (may or may not be correct).
Two explanations for the Answer.
A couple of notes:
- You may disagree with the Answer, but you should pretend it is correct when selecting the best Explanation.
- You can ignore minor grammar and spelling mistakes. Some sentences will be lowercased incorrectly; please ignore this.
- If you find the Explanations similarly good (or bad), select the one you think is the best and then select "I voted, but this one is close because the explanations are similar."
- If you find the Explanations equally good (or bad) and cannot choose, select "Both equally good." or "Both equally bad."
Examples (click to expand/collapse)
Question: ${question}
Answer: ${gold_label}
Which explanation better explains the answer?
${Explanation_1}
${explanation_2}
Both equally good.
Both equally bad.
I voted, but this one is close because the explanations are similar.
(Optional) Please let us know if anything was unclear, if you experienced any issues, or if you have any other feedback for us.
Figure 5: An overview of the user interface of our head-to-head comparison AMT studies for CommonsenseQA. The top shows the instructions and the bottom the actual task. The Examples tab is collapsed here; shown in full in Figure 6.
Question: If a lantern is not for sale, where is it likely to be?
Answer: house
A house has lanterns.
people keep furniture, such as lanterns, at their houses for personal use in their daily living.
The better explanation is
people keep furniture, such as lanterns, at their houses for personal use in their daily living. . It explains why
"house" answers the part of the question that specifies "not for sale", which the first choice does not explain.
Question: "there are 10 apples on an apple tree. three fall off. now there are x apples." what is this an example of?
Answer: math problem
The example is of a question that involves performing subtraction.
webmath is designed to help you solve
The better explanation is The example is of a question that involves performing subtraction. The second explanation is nonsensical.
Example #3
Question: Where is the most likely place for more than one stadium?
Answer: large city
a stadium requires a lot of space.
a good place for more than one stadium is a place with a lot of people.
The better explanation is a good place for more than one stadium is a place with a lot of people. It explains why a large city would have multiple stadiums. The first explanation is factually correct, but does not explain, for example, why a large city would have more space than a rural area.
Figure 6: The Examples tab given in the user interface of our head-to-head comparison AMT studies for Common-senseQA. The full interface is shown in Figure 5.
You will read multiple-choice questions about daily life. For each question, an answer has been selected and a statement explaining the answer has been given.
Part 1: Please read the statement and answer questions related to its factuality and grammaticality (2 questions).
Part 2: Please read the instance that the statement explains and answer questions about the explanation quality (1-3 questions).
An instance contains 3 parts:
The statement from Part 1, which explains the Answer.
[Important!] You may disagree with the Answer, but you should pretend it is correct when judging the Explanation.
[Important!] Some sentences will be lowercased incorrectly; please ignore this.
[Important!] You will not be able to change your answers to Part 1 after moving on to Part 2.
Main Example (click to expand/collapse)
Part 1: Please read the below statement and answer the following questions:
{$predicted Explanation}
1) How factual is this statement?
Generally False
Sometimes or Partially True
Generally True
Needs More Information to Judge
2) Is this statement grammatical?
No Yes
Submit Part 1
Figure 7: An overview of the user interface template of our absolute comparison AMT studies for Common-senseQA. The top shows the instructions and the bottom the actual task. Only part 1 of the task is shown here (part 2 appears once part 1 is submitted). The Main Example and More Examples tabs illustrating both parts 1 and 2 are collapsed here; see Figure 8.
Generally False Sometimes or Partially True Generally True Need More Information to Judge
This statement is Generally True
The correct answer is Yes.
Part 2: Please read the instance that the statement explains and answer the following questions:
Explanation: people keep furniture, such as lanterns, at their houses for personal use in their daily living.
3) Does the Explanation provide new facts, information, or reasoning not stated in the Question and Answer?
The correct answer is Yes, because the explanation introduces the fact that lanterns are a type of furniture, and that people do not usually keep furniture in their house for selling.
4) Is the new information relevant to justifying the Answer? Examples
The correct answer is
5) How much information does the Explanation have to justify the Answer? Examples
Figure 8: The Main Example given in the user interface template of our absolute comparison AMT studies for CommonsenseQA. This format follows the actual task layout. The full interface is shown in Figure 7.
You will read questions about daily life. For each question, an answer has been selected and 5 explanations have been given.
Your task is to judge whether each explanation is acceptable.
Five explanations for the Answer.
- You may disagree with the Answer, but you should pretend it is correct when judging the Explanations.
- Some sentences have minor grammar and spelling mistakes, don't end on a period, or cut off partway through because they are long. Please ignore these, and judge on the part of the sentence that is presented.
- New! If at least part of the explanation is acceptable and sufficient (even if some other part is nonsensical or rambling), please mark is as acceptable.
Examples (New!) (click to expand/collapse)
Which explanations are acceptable? Select all that you believe adequately explain the answer (this may be none).
${explanation_1}
{\explanation_2}
{$explanation_3}
${explanation_4}
${explanation_5}
Figure 9: An overview of the user interface of our explanation acceptability AMT studies for CommonsenseQA. The top shows the instructions and the bottom the actual task. The "examples" tab is collapsed here; shown in full in Figure 10.
Examples (New) (click to expand/collapse)
Question: A bald eagle flies over St. Paul, where is it?
Answer: minnesota
Bald eagles are native to the United States, but are rarer in Minnesota than in other states because Minnesota has a small supply of their primary foods (fish, mice, etc.).
St. Paul is a city in Minnesota, so a bald eagle flying over it would likely live in Minnesota.
St. Paul is in Minnesota.
The question assumes that you've seen the bald eagles that live in Minnesota.
Bald eagles live in northern North America, and can be found anywhere there. Given that one is flying over St. Paul, it must be either migrating, or it could be a photographer, making its appearance in the city unusual.
Possible answer:
One possible answer is that only the second and third explanations are acceptable. The first, fourth, and fifth explanations do not provide an explanation for the fact that St. Paul is in Minnesota.
Another possible answer is that only the third explanation is acceptable (since the second explanation incorrectly states that the bald eagle lives in Minnesota). However, we prefer that you err on the side of being forgiving if part of the explanation is correct.
Question: Even if was clean and chlorinated, shallow water wouldn't make for much summer fun in
your backyard what? Answer: swimming pool
- It would be illegal to use and non-sanitary to use a lake or other body of water as a swimming pool.
Pools seem to be the kind of shallow water that people would swim in or fun.
Swimming pools get deeper than puddles or ponds.
- Pools are the only body of water that contain both swimming-friendly water and shallow water.
Swimming pools are often used for summer fun.
It would be illegal to use and non-sanitary to use a lake or other body of water as a swimming pool.
Pools seem to be the kind of shallow water that people would swim in for fun.
Pools are the only body of water that contain both swimming-friendly water and shallow water.
The question implies that the answer must be a body of water that can get deep, and must be able to be used for summer fun.
One possible answer is that only the fifth explanation is acceptable. The first, second, and fourth explanations do not explain that pools can get deep or that pools can be used for summer fun. The fifth explanation explains at least one of these facts.
Another possible answer is that both the third and the fifth explanations are acceptable, since the third explanation indirectly states that pools can get deep (although it is somewhat debatable if they are deeper than ponds).
Figure 10: The examples given in the user interface of our explanation acceptability AMT studies for Common-senseQA. The full interface is shown in Figure 9.