SlowGuess/NAACL_2022_OCR huggingface dataset:
Sarah Wiegreffe\* Jack Hessel' Swabha Swayamdipta Mark Riedl' Yejin Choi
$\clubsuit$ School of Interactive Computing, Georgia Institute of Technology †Allen Institute for Artificial Intelligence
$\diamond$ Paul G. Allen School of Computer Science and Engineering, University of Washington saw@gatech.edu, riedl@cc.gatech.edu, {jackh, swabhas, yejinc} @allenai.org
Large language models are increasingly capable of generating fluent-appearing text with relatively little task-specific supervision. But can these models accurately explain classification decisions? We consider the task of generating free-text explanations using human-written examples in a few-shot manner. We find that (1) authoring higher quality prompts results in higher quality generations; and (2) surprisingly, in a head-to-head comparison, crowdworkers often prefer explanations generated by GPT-3 to crowdsourced explanations in existing datasets. Our human studies also show, however, that while models often produce factual, grammatical, and sufficient explanations, they have room to improve along axes such as providing novel information and supporting the label. We create a pipeline that combines GPT-3 with a supervised filter that incorporates binary acceptability judgments from humans in the loop. Despite the intrinsic subjectivity of acceptability judgments, we demonstrate that acceptability is partially correlated with various fine-grained attributes of explanations. Our approach is able to consistently filter GPT-3-generated explanations deemed acceptable by humans.
As natural language understanding tasks have become increasingly complex, the field of explainable NLP has embraced explanations written in freeform natural language. In contrast to extractive explanations that highlight tokens in the input, fre-text explanations provide a natural interface between machine computation and human end-users (Hendricks et al., 2016; Camburu et al., 2018). The dominant paradigm for producing free-text explanations is via direct supervision, i.e., training an autoregressive, generative language model to predict human-authored explanations directly (Kim et al., 2018; Park et al., 2018; Ehsan et al., 2018; Narang et al., 2020; Wegreffe et al., 2021, i.a.).
Figure 1: Illustration of our overgeneration + filtration pipeline for producing human acceptable explanations for CommonsenseQA and SNLI (see examples in Table 1). Authors of this work write explanations to prompt GPT-3, generating 5 explanations per instance. An acceptability filter, trained on human binary acceptability judgments, determines which of these generated explanations are plausible. Evaluation is performed at both the explanation and the instance level.
However, collecting high-quality written explanations to serve as supervision is difficult and expensive. More than  $70\%$  of existing free-text explanation datasets are crowdsourced (Wiegreffe and Marasovic, 2021), and even the most meticulous crowdsourcing efforts frequently fail to elicit logically consistent and grammatical explanations (Narang et al., 2020). Furthermore, a lack of standardized crowdsourcing design has resulted in highly varied datasets, which are hard to compare or combine (Tan, 2021).
Human Language Technologies, pages 632 - 658
Recent progress in prompting large language models (LLMs) provides a potentially promising alternate to large-scale crowdsourcing. The in-context learning paradigm, wherein powerful language models are prompted in a few-shot manner with just a few examples, has proven surprisingly effective across a range of NLP tasks (Radford et al., 2019; Brown et al., 2020; Shin et al., 2020; Schick and Schütze, 2021a, i.a.). In this work we ask: can LLMs also generate reliable explanations? In human subjects studies, we find that GPT-3 (Brown et al., 2020) can be readily made to generate explanations via prompting, and surprisingly, humans often prefer GPT-3 generated explanations to crowdsourced explanations in existing datasets ( $\S 2$ ).
Two additional human subjects studies, however, demonstrate that GPT-3-generated explanations still have significant room for improvement along axes such as providing new information (i.e., avoiding repetition) and supporting the label; human subjects found less than half of greedily-decoded GPT-3-generated explanations to be acceptable with  $100\%$  agreement. To improve upon this, we re-frame the role of crowd annotators: instead of asking them to write explanations as in prior work, we (1) repeatedly query GPT-3 to generate multiple candidate explanations for each input instance, and (2) ask crowdworkers to rate the acceptability of each candidate generation. After showing that GPT-3 can usually generate an explanation that humans unanimously find acceptable within as few as five queries ( $\S 3$ ), we use a small number of these binary crowdworker judgments to supervise an acceptability filtering model, which can be applied to select high quality candidates among GPT-3's outputs (Figure 1;  $\S 4$ ).
Despite intrinsic subjectivity in acceptability ratings, our supervised model improves upon the already-competitive few-shot paradigm by consistently selecting (human-identified) high quality explanations better than strong baselines. Human evaluations reveal that the filtration model not only improves acceptability, but also other axes like supporting the label and providing novel information.
In summary, our main findings are:
i. few-shot prompting with GPT-3 produces surprisingly competitive explanations, providing a promising alternative to crowd-authored free-text explanation corpora;
ii. binary human labeling can instead be leveraged
to train a filter that selects high-quality machine-generated explanations; and
iii. in areas where GPT-3 struggles, including information content, supporting the label, and overall acceptability, our proposed overgenerate-and-filter pipeline improves generated explanations. We publicly release our code and data.<sup>1</sup>
2 GPT-3 is Competitive with Crowdsourced Explanation Datasets
We investigate three research questions:
1. Are GPT-3-generated explanations preferable to crowdsourced ones in existing datasets? (§2.1)
2. Can improving prompt quality improve GPT-3-generated explanations? (§2.2)
3. Along what fine-grained dimensions are GPT-3-generated explanations preferred, and do these correlate with overall acceptability? (§2.3)
Explanation tasks and datasets. We consider two English tasks: CommonsenseQA and natural language inference (NLI), shown in Table 1. CommonsenseQA (Talmor et al., 2019) is a multiple choice task posed over commonsense questions. Crowdsourced free-text explanations for instances in CommonsenseQA are provided in the CoS-E v1.11 (Rajani et al., 2019) and ECQA (Aggarwal et al., 2021) datasets. ECQA explanations are counterfactual, i.e., annotators were instructed to explain not only the correct answer choice but also why the others are incorrect. ECQA was released to address the quality issues of CoS-E (Narang et al., 2020); for completeness, we experiment with both. Our second task is NLI, which involves inferring whether a given hypothesis sentence entails, contradicts, or is neutral towards a premise. This task is instantiated with the SNLI dataset (Bowman et al., 2015) and crowdsourced explanations from e-SNLI (Camburu et al., 2018). For each task, we report results on a fixed, randomly-sampled 250-instance test set not observed during prompt design.
Few-shot prompting for explanations. We use GPT-3 Davinci $^2$  (Brown et al., 2020), an autoregressive language model with  $\sim 175\mathrm{B}$  parameters trained on a large dataset of text scraped from the internet. We prompt the model with several (question, gold answer, explanation) triplets followed by an unexplained question-gold answer instance for
<sup>1</sup>https://github.com/allenai/few_shot_explanations/
2Non-instruct version, available in Summer 2021.
SNLI (Bowman et al., 2015)
Premise: Dark-haired man wearing a watch and oven mitt about to cook some meat in the kitchen.
Hypothesis: A man is cooking something to eat.
Label: entailment
e-SNLI (Camburu et al., 2018): Meat is cooked in a kitchen, and is a food that you eat. Using an oven mitt implies you're about to cook with hot utensils.
GPT-3: Cooking is usually done to prepare food to eat.
CommonsenseQA (Talmor et al., 2019)
Question: What is the result of applying for job?
Answer Choices: anxiety and fear, increased workload, praise, less sleep, or being employed
Correct Choice: being employed
CoS-E (Rajani et al., 2019): being employed applying for job
ECQA (Aggarwal et al., 2021): Applying for a job is followed by attending interview which results in being employed. Applying for a job may not result in the other options.
GPT-3: Applying for a job can result in being employed, which is a positive outcome.
Table 1: Task-specific instances, along with their crowdsourced explanations from the respective datasets, shown alongside explanations generated greedily by GPT-3. In our experiments, the SNLI GPT-3 explanation was preferred over its corresponding e-SNLI explanation by  $2/3$  annotators. For CommonSenseQA,  $3/3$  preferred the GPT-3 explanation to the CoS-E explanation, and  $2/3$  to the ECQA one.
which we expect the model to generate an explanation. $^3$  We use a total of 115 randomly sampled train instances to create our prompts; each prompt consists of 8-24 randomly selected examples from this set. For each instance, we generate a single explanation with greedy decoding. More details about prompt construction are in Appendix A; example prompts are given in Tables 2 and 11.
Crowdsourcing evaluation. Given that existing automatic metrics often do not correlate well with human judgements of explanation quality (Clinci et al., 2021; Kayser et al., 2021), we conduct human
Let's explain classification decisions.
A young boy wearing a tank-top is climbing a tree.  
question: A boy was showing off for a girl.
true, false, or neither? neither
why? A boy might climb a tree to show off for a girl, but he also might do it for fun or for other reasons.
question: A person is outdoors, on a horse.
true, false, or neither? true
why? Horse riding is an activity almost always done outdoors. Additionally, a plane is a large object and is most likely to be found outdoors.
question: The horses are becoming suspicious of my apples.
true, false, or neither? false
why? The presence of a red truck does not imply there are apples, nor does it imply the horses are suspicious.
question: A dog is asleep in its dog house.
Table 2: Example of a prompt with 3 training examples for SNLI: presented are the premise/hypothesis pairs, the gold labels, and the explanations (written by us) that act as input to GPT-3 (in practice, we use 8-24 examples per prompt). The text generated by the model acts as the free-text explanation. In this case, the model greedily auto-completes (given 12 examples): "A dog cannot carry something while asleep".
studies for evaluation.4 We ensure each experiment has a substantial number of distinct crowdworkers to mitigate individual annotator bias (Table 17).
We present workers with a dataset instance, its gold label, and two explanations for the instance generated under different conditions ("head-to-head"). We then ask them to make a preferential selection, collecting 3 annotations per data point. We report inter-annotator agreement (IAA) using Krippendorff's  $\alpha$  (Krippendorff, 2011). We find low-to-moderate agreement across studies, indicating the subjective nature of the task; see Tables 3, 4, and 5. Appendix B contains further details on quality control.
3We condition on the gold label as a methodological control to ensure reliable human evaluation. In pilot studies, we found it hard to avoid bias against explanations when we disagreed with the predicted label. Prior work (Kayser et al., 2021; Marasovic et al., 2021) has removed this confounder by only considering explanations for correctly-predicted instances, which may overestimate explanation quality. Our method allows us to report results on a truly representative sample. We experiment with incorrect vs. correct predictions in Appendix C, finding that GPT-3 can competitively explain gold labels for instances it predicted incorrectly.
Our studies were conducted on the Amazon Mechanical Turk platform, at the Allen Institute for AI. We selected workers located in Australia, Canada, New Zealand, the UK, or the US, with a past HIT approval rate of  $>98\%$  and  $>5000$  HITs approved, and compensated them at a rate of $15/hour. Each worker completed qualifying exams on explanation evaluation and the NLI task.
Dataset	Preferred Explanation (%)	α
Crowd	Tie	GPT-3
CoS-E	20.3	34.9	44.8	0.5
ECQA	52.7	12.9	34.4	0.2
e-SNLI	63.6	11.6	24.8	0.3
2.1 Are GPT-3 explanations preferred over crowdsourced ones?
We perform a head-to-head comparison of explanations generated by GPT-3 with greedy decoding vs. gold human-written explanations in the original datasets. The crowdsourced explanations serve as a reasonable upper bound for what a supervised explanation generation model trained on them could produce. Table 1 contains examples of GPT-3-preferred explanations.
Results are shown in Table 3. GPT-3 greedily-decoded explanations are frequently preferred or comparable to crowdsourced explanations in CoSE, which is not too surprising given the dataset has many ungrammatical explanations (Narang et al., 2020). And, while ECQA and e-SNLI explanations are strongly preferred to GPT-3, there are still a nontrivial number of cases where GPT-3 explanations are competitive (47.3% and 36.4%, respectively).
2.2 Can improving prompt quality improve GPT-3-generated explanations?
Table 3: Head-to-head human evaluation for 250 explanations generated by GPT-3 vs. written by crowd-workers in the datasets, along with Krippendorff's  $\alpha$  for IAA. Results are shown as  $\%$  preferences. We prompted GPT-3 with crowd explanations from the corresponding datasets.
Dataset	Preferred GPT-3 Explanation (%)	α
Crowd Prompts	Tie	Our Prompts
CoS-E	6.9	13.5	79.6	0.2
ECQA	15.9	9.5	74.7	0.3
e-SNLI	30.8	26.8	42.4	0.5
Given that low-quality training instances may result in low-quality predictions (especially in a few shot setting), we ask: can we improve GPT-3 gen
Table 4: Head-to-head human evaluation for 250 explanations generated by GPT-3 prompted with either author-written explanations or crowdsourced explanations from the associated datasets, along with Krippendorff's  $\alpha$  for IAA.
Dataset	Preferred Explanation (%)	α
Crowd	Tie	GPT-3
CoS-E	7.2	13.9	78.9	0.5
ECQA	44.5	9.7	45.7	0.4
e-SNLI	49.6	9.7	40.7	0.2
Table 5: Head-to-head human evaluation for 250 explanations generated by GPT-3 vs. written by crowd-workers in the datasets, along with Krippendorff's  $\alpha$  for IAA. GPT-3 explanations were prompted with explanations written by the authors.
erations simply by conditioning on higher-quality instances? To this end, we replace the 115 crowdsourced explanations from the original datasets for prompting GPT-3 with explanations carefully written by the authors of this paper (see Table 12 for examples). Our prompts are used to generate a different set of GPT-3 explanations on the same test data.
We perform a head-to-head human evaluation of the GPT-3 generations conditioned on the explanations we wrote vs. those conditioned on the crowdsourced explanations. Results in Table 4 show that, for all three corpora, generations conditioned on our explanations outperform generations conditioned on crowdsourced ones, illustrating the importance of good-quality prompts for GPT-3.
We repeat the experiment of §2.1, but with our prompts instead of dataset prompts. With this change, GPT-3 generations are even more competitive (Table 5). For all three datasets, more than half the time, few-shot prompting results in an explanation at least as good as a human-written explanation. For subsequent experiments, we prompt GPT-3 with the author-written explanations.
2.3 What types of explanations does GPT-3 generate?
Pairwise evaluations can only offer perspective on the relative quality of generated explanations. Are crowd annotators simply comparing explanations on surface-level features like grammaticality?
To understand finer-grained characteristics of explanations, we design a second human study to collect absolute Likert-scale judgments across seven axes of quality (with each explanation judged by 3 annotators). The first three axes capture surface-level features: generality, grammaticality, and factuality. The next three capture richer aspects of explanation quality: whether new information is introduced (a requirement for non-vacuous explana
For example, GPT-3 reproduces known data artifacts in the CoS-E corpus when prompted with explanations from it, such as the recurring phrase "rivers flow trough (sp) valleys".
Figure 2: Absolute evaluation for GPT-3 and crowdsourced explanations for CommonsenseQA via CoS-E (top) and ECQA (middle) datasets, and for NLI via the e-SNLI dataset (bottom). The distribution of mean scores of 3 annotators for each instance in the test set is plotted. All attributes besides Factuality and Amount Info are binary. *Amount Info is the only attribute for which a value of 0 is preferred to a value of 1. See Table 18 for more details.
tions), whether explanations support the gold label, and whether the amount of information given is sufficient. Finally, we ask for an overall judgement of quality: is the explanation acceptable? We explain our design process in Appendix B.3. Results on the crowdsourced and GPT-3 explanations for both tasks are given in Figure 2. $^6$
For both tasks, GPT-3 explanations do well in all 3 surface-level categories, with statistically significantly greater ratings in generality and grammaticality (and factuality for CommonsenseQA) compared to crowdsourced explanations, and distributional means close to 1. In these categories, there is little room for improvement.
On the other hand, GPT-3 explanations do not contain as much new information as ECQA and e-SNLI explanations, indicating substantial room for improvement (mean=0.1 for both tasks compared to 0.6 for ECQA and 0.2 for SNLI; these differences are statistically significant at  $p \leq 0.01$ ). GPT-3 explanations are substantially more supportive of the label vs. CoS-E, but not as supportive as ECQA or e-SNLI (all statistically significant at  $p \leq 0.1$ ). Indeed, the mean rating of GPT-3 explanations for label support is 0.5 for Common
senseQA and  $-0.1$  for NLI, demonstrating room for improvement. These axes are crucial to ensuring explanations are not vacuous and are on-topic. Finally, GPT-3 explanations are judged as acceptable at higher rates than CoS-E or ECQA explanations, but not e-SNLI explanations. Mean scores of 0.5 (CommonsenseQA) and 0.0 (NLI) indicate that GPT-3 explanations have room to improve overall. $^7$
Correlation between acceptability and other attributes To understand what factors are important for the overall "acceptability" judgement, we compute Spearman correlation  $(\rho ;$  Spearman, 1987) between acceptability and all other attributes (Table 6). Each is positively correlated with acceptability, though with varying degrees of magnitude. Acceptability is least correlated with "new information," and most correlated with grammaticality, generality, and the explanation's support for the label. Overall, the results indicate that human preference for explanations is not fully explained by any one attribute, and is not limited to surface-level features.
<sup>7</sup>We do not evaluate explanations counterfactually, which may explain why ECQA explanations are often labeled as having "too much information". See Appendix B.1.
$^{6}$ Krippendorff's  $\alpha$  is 0.48 for CommonsenseQA annotations and 0.31 for SNLI; see Table 15.
Attribute	ρ	n
Generality	0.25	3750
Factuality	0.17	2958
Grammar	0.31	3750
New Info	0.05	3750
Supports Label	0.22	2943
*Amount Info	0.16	1761
Table 6: Spearman's correlation  $(\rho)$  between acceptability and other attributes in Figure 2. For Amount Info, a value of 0 is preferred to a value of 1. All results are statistically significant at  $p < 0.01$
3 Beyond Greedy Explanations
While GPT-3 explanations demonstrate strength across surface-level features and are surprisingly competitive in head-to-head settings, they can still be improved. One might imagine a setting with multiple end-users in which we wish to provide the most unambiguously acceptable explanation as output from a system. When considering the data from §2.3, we find that only  $46.3\%$  of the greedily-decoded GPT-3-generated explanations for CommonsenseQA and  $31.5\%$  for NLI are rated acceptable by 3/3 annotators. $^{8}$
Inspired by work in other generation tasks (Holtzman et al., 2020; Massarelli et al., 2020; Holtzman et al., 2021), we hypothesize that equally or more informative explanations can be generated by sampling stochastically. We sample 4 additional generations from GPT-3 for each instance to complement the greedy generation. We crowdsourced 3 acceptability annotations for each new explanation.
As expected, sampled explanations exhibit lower  $3/3$  acceptability than greedy explanations (25.1% for CommonsenseQA; 11.3% for SNLI). However, this results in a surprisingly higher proportion of instances that have at least one acceptable explanation in the set of 5. The greedy explanation was judged to be  $3/3$  acceptable in 46.3% of instances for CommonsenseQA and 31.5% for NLI; this increases to 79.5% and 51.2%, respectively, when sampled explanations are included.[9]
4 Improving Explanation Generation with Acceptability Filtering
The challenge of overgeneration is that GPT-3 alone cannot discern which of its stochastic samples
are acceptable. Inspired by West et al. (2021), we explore training a supervised filter on the collected labels. Our key intuition is that by re-framing the role of annotators from explanation authors to binary judges, we can alleviate the need to collect a large-scale explanations dataset—the result is a simpler, cheaper, and easier crowdsourcing setup to administer ( $\S 4.1$ ). Namely, we can (1) aggregate ratings over multiple annotators to produce more reliable labels, (2) use numerical metrics of annotator agreement to remove annotators contributing noisy data, and (3) collect annotations more quickly and cheaply than asking annotators to hand-write explanations. Moreover, we find that the filter can be trained with a relatively small amount of binary human judgments ( $\S 4.2$ ). We demonstrate that the trained model is not simply taking advantage of surface-level features ( $\S 4.4$ ). Figure 1 presents an overview of our pipeline.
4.1 Acceptability Annotations
We generate train and validation sets by repeating the procedure of generating 1 greedy and 4 sampled explanations for 991 and 1K instances, respectively, of the CommonsenseQA and SNLI training sets. Combining these with the annotated test sets from previous experiments results in a dataset of 1241/1250 instances in a  $72/8/20\%$  train/val/test ratio for each task. We again collect 3 binary acceptability ratings for each instance, resulting in  $\sim 6200$  instance-explanation pairs and  $\sim 19\mathrm{K}$  individual annotations per task. Table 13 contains statistics. To ensure that models trained on these corpora do not overfit to specific annotators (Geva et al., 2019), we collect an additional set of judgments for the test set of SNLI from a group of annotators who did not participate in any of our previous annotation tasks ("Test2"). Figure 9 and Figure 10 show the user interface. $^{10}$
While we evaluate at test-time with the schema that only instances that  $3/3$  annotators deem acceptable are considered acceptable, preliminary experiments show that treating both  $2/3$  and  $3/3$  agreement instances as acceptable during training performs best on the  $3/3$  evaluation criterion at test-time.[11] We also train a variant where we randomly select one annotation from the three as the gold label ("without human agreement").
$^{8}$ In  $\S 4$ , we show that these are not upper-bounds caused by intrinsic subjectivity, and that they can be improved upon.  
 $^{9}$ Appendix D provides similar results for the  $2/3$  acceptability threshold.