SlowGuess/NAACL_2022_OCR huggingface dataset:
$^{10}$ Krippendorff's  $\alpha$  for all acceptability annotations is 0.34 for CommonsenseQA and 0.39 for SNLI (see Table 14). Our results don't significantly change if a  $2/3$  cutoff is used at test time instead: Appendix F contains the results.
4.2 Acceptability Filter
Concretely, given the problem instance, the gold label, and a generated explanation, the acceptability filter predicts whether the explanation is acceptable. We fine-tune two sequence-to-sequence architectures, T5-Large (Raffel et al., 2020) and T0-3B (Sanh et al., 2022). Each model is trained  $5 \times 5$  with different random seeds. Further training details are given in Appendix E.
Baselines. We train an explanation-only baseline, which receives as input only the explanation; similar baselines have been proposed for NLI (Poliak et al., 2018; Gururangan et al., 2018). These models represent the hypothesis that annotator ratings can be reconstructed with only surface features of the explanation candidates, e.g., grammaticality. We also consider a negative log-likelihood (NLL) baseline, which uses GPT-3's estimated probability as the acceptability classification score. This is a slightly more competitive baseline than greedy; greedy usually (but not always) produces the highest-likelihood explanation. $^{12}$
We consider three evaluation settings. The first is instance-level ("select-1"), where the system returns 1 explanation selected from the set of 5 for each instance. We return the explanation with the highest model-estimated probability and report instance-level accuracy, i.e., the  $\%$  of instances for which a gold acceptable explanation is selected.
We also evaluate at the explanation-level, where we treat each explanation independently and compute metrics over the full dataset. This aligns with the binary classification training of the models (cross-entropy on the explanation labels) and is suited for the setting in which we want to return all of the acceptable explanations per instance. In this setting, we report average precision (AP), an estimate of area under the precision-recall curve.
Finally, we perform an absolute human evaluation (§2.3) on the subset of instances where the filter model does not select the greedy explanation as the best, i.e., comparing "select-1" performance to a greedy baseline on the instances where it differs. We additionally re-perform the head-to-head comparison of Table 5, replacing the greedy GPT-3 explanations with those selected by the filter.
“Select-1” Acc@3/3	Expl.-level AP@3/3
Dev	Test	Dev	Test
Random	26.83.2	30.62.1	27.41.1	31.60.4
Constant	—	—	27.9	31.8
NLL	41.8	52.0	42.4	45.6
T5-L Expl.-only	40.23.9	49.81.1	43.51.5	50.01.9
T0-3B Expl.-only	42.61.4	47.31.6	41.12.0	54.11.7
T5-L w/o HA	46.62.3	55.43.2	47.02.3	56.93.6
T5-L	46.42.9	55.42.1	45.13.4	58.31.5
T0-3B w/o HA	48.42.0	57.42.8	44.52.3	59.82.7
T0-3B	48.60.9	59.91.1	49.71.6	64.01.5
Oracle U.B.	78.0	82.0	100.0	100.0
Table 7: Results for acceptability classifiers trained on CommonsenseQA. Subscripts indicate standard error over models trained with 5 different random seeds. "w/o HA" = without human agreement. "Oracle U.B" indicates upper bound based on dataset properties (§3).
“Select-1” Acc@3/3	Explanation-level AP@3/3
Dev	Test	Test2	Dev	Test	Test2
Random	15.20.7	14.70.1	13.60.3	15.00.6	14.40.3	13.80.2
Constant	—	—	—	15.2	14.5	13.7
NLL	33.0	32.0	31.6	29.9	32.7	28.5
Expl.-only	30.20.8	30.92.1	27.81.8	30.62.5	30.61.3	25.92.3
w/o HA	38.21.9	38.51.8	36.21.4	49.35.3	48.53.3	52.85.3
Full	37.80.8	38.70.7	35.01.2	46.83.6	47.63.5	49.54.8
Oracle U.B.	51.0	52.4	46.4	100.0	100.0	100.0
Table 8: Results for SNLI explanation acceptability; all model results are on T0-3B. See Table 7's caption.
Classifier performance is given in Tables 7-8.
Effect of model size. On CommonsenseQA, T0-3B outperforms T5-Large by  $\sim 2 - 4\%$  select-1 accuracy and  $\sim 5 - 6\%$  explanation-level AP across splits. We use T0-3B in subsequent experiments.
NLL baseline vs. full model. For both tasks on both validation and test sets, T0-3B outperforms the NLL baseline substantially. On CommonsenseQA, we observe a  $7 - 8\%$  gain in instance-level accuracy, and a gain of  $18\%$  explanation-level AP on the test set. This provides strong evidence that the supervised model is able to incorporate binary human feedback to predict acceptable explanations at a level much higher than GPT-3 achieves on its own. We present examples where "select-1" predictions differ between NLL and our filter model in Table 10 and Table 16.
Explanation only vs. full model. Our results suggest that our models are leveraging feature interactions between the instance and explanation to make their predictions: without instance-level context, the explanation-only baselines are on average more than 5 points worse across metrics. Though
12 According to GPT-3, a sampled explanation from the set of 5 has a lower NLL than greedy for only  $2.8\%$  and  $3.6\%$  of instances, respectively, of our Com.QA and SNLI test sets.
Figure 3: Absolute evaluation results in the "select-1" setting for the instances where our best-performing filter model does not select the greedy explanation (156 instances for CommonsenseQA (top); 91 for NLI (bottom)). See caption of Figure 2 and the Appendix-Table 20 for more details and statistical significance results.
Dataset	Preferred Explanation (%)	α
Gold-Standard	Tie	GPT-3
CoS-E	2.9	3.6	93.5	0.25
ECQA	19.9	10.0	70.1	0.05
e-SNLI	49.9	6.9	43.2	0.14
Table 9: A replication of Table 5, except the GPT-3 explanations are now the top-1 of our filter system.
they under-perform significantly relative to the full model, explanation-only baselines do fare surprisingly well, indicating that shallow features like factuality and grammaticality may represent latent factors in human acceptability judgments.
The effect of multiple training annotations. In some cases, performance improves if the training instances are labeled with the consensus of three annotators (vs. the singularly annotated case "w/o HA"), though the effects are inconsistent. In most cases, using consensus labels results in reduced variance across random seeds. However, the gains may not outweigh the 3x annotations required.
Our model doesn't overfit to specific annotators. The performance of our model when evaluated on the NLI test set labeled by separate annotators ("Test2") is comparable to the original test set (instance-level accuracy drops a few points, but explanation-level AP slightly rises).
Our model improves generated explanations along desirable traits. We present our absolute human evaluation for greedy vs. filtered explanations from GPT-3 in Figure 3— for both tasks, explanations filtered by our model more readily
SNLI
Premise: An officer in a black uniform and hat stands to the left of a large structure with other officers in the background.
Hypothesis: An officer enjoys coffee in a shop.
Label: contradiction
NLL-Predicted Explanation: An officer in a black uniform and hat is not necessarily an officer enjoying coffee in a shop.
Filter-Predicted Explanation: An officer in a structure standing to one side is not the same as enjoying coffee in a shop.
Question: Where would there be an auditorium with only a single person speaking?
Answer choices: theater, park, university campus, crowd, or lights
NLL-Predicted Explanation: An auditorium is a large room used for lectures, and a single person speaking is likely to be a lecture.
Filter-Predicted Explanation: On university or university-like campuses, the auditoriums are often used for lectures or other University events, where a single person might be speaking.
Table 10: Randomly-selected instances that our filter model predicts differently than NLL at the "select-1" task and got correct, but NLL got incorrect.
introduce new information, support the label, and contain at least enough information for both tasks (in addition to being more acceptable). Interestingly, greedy explanations still prevail in surface-level features (grammaticality and, in the case of CommonsenseQA, factuality; differences are statistically significant with low  $p$ , see Table 20).<sup>13</sup>
We additionally find in our head-to-head study
$^{13}$ Krippendorff's  $\alpha$  for these experiments is 0.32 for CommonsenseQA and 0.33 for SNLI.
(Table 9) that, compared to Table 5, using filtered GPT-3 explanations instead of greedy increases the preference for GPT-3 explanations by  $15 - 24\%$  for both CommonsenseQA datasets. We do not see an increase in the SNLI case, which may be due to the fact that fewer GPT-3 explanations change after filtering  $(36.4\%)$ , compared to  $62.4\%$  for CommonsenseQA), and GPT-3 explanations for SNLI tend to be less acceptable overall, resulting in a lower upper-bound oracle of instances where an acceptable explanation can be selected (§3).
In summary. We have demonstrated the effectiveness of modeling binary crowd judgements of acceptability as a means to select candidates from GPT-3 which are deemed acceptable with unanimous agreement. For the method that does not leverage human agreement, this is done with only  $\sim 5\mathrm{k}$  binary annotations. We additionally demonstrate that our filtered explanations improve upon greedy generations in fine-grained categories that probe their topical relevance and meaningful content. The gap between our best model and the upper-bound oracle indicates that there is still substantial room for improvement in both task settings (but especially for SNLI). Future work may investigate sampling more explanations, or incorporating other sources of supervision signal.
Free-text explanation generation. The earliest neural free-text explanation models did so for computer vision applications (Hendricks et al., 2016; Park et al., 2018; Kim et al., 2018) and NLI (Camburu et al., 2018). These methods relied on supervised datasets to train the explanation generator. Others have proposed to generate explanations or clarifications to improve task performance in a supervised (Rajani et al., 2019; Lampinen et al., 2022) or unsupervised (Shwartz et al., 2020) manner. Yordanov et al. (2021) study transfer learning between datasets for few-shot generation.
Latcinnik and Berant (2020) proposed a method to generate free-text explanations supervised only on task signal, and Brahman et al. (2021) used sources of weak supervision to generate explanations for defeasible inference. Paranjape et al. (2021) design hand-crafted templates which they use with mask-infilling to produce contrastive explanations from pretrained language models.
Concurrent work (Marasović et al., 2021) studies the effect of prompt format and model size on
crowdworker judgements of prompted explanation plausibility. They find that GPT-3 Davinci outperforms other smaller pretrained models, but that crowdworkers find these explanations less plausible than those from the datasets, aligning with our first experimental result (Table 3). We perform a more in-depth study of the fine-grained criteria comprising human acceptability, and demonstrate that with higher-quality prompts and filtering, GPT-3's performance can be significantly improved.
Supervising on human preferences. Prior and concurrent work has used binary judgements from crowdworkers to fit models to human preferences for non-XAI tasks such as summarization (Ziegler et al., 2020; Stiannon et al., 2020), creating commonsense knowledge bases (West et al., 2021), and building natural language inference datasets (Liu et al., 2022). Unlike these works, we apply human preference modeling to increase the human acceptability of model-generated free-text explanations. West et al. (2021) demonstrate that GPT-3 + a supervised acceptability filter can generate a high-quality causal knowledge graph: in addition to their work being conducted in a different domain, our success conditions and evaluation metrics differ because we must produce a prediction for each instance (whereas they can simply discard bad generations).
We demonstrate GPT-3's capacity to generate fre-text explanations for NLP task instances in a few-shot setting. We further improve this capability via an overgenerate + filter approach, where the filter is trained on supervision from human acceptability ratings. We hope our results can guide future work on free-text explanations via neural or neuro-symbolic systems (Brahman et al., 2021; Majumder et al., 2021; Saha et al., 2021). Future work may also further investigate the benefits of counterfactual explanations.
While human rationales for decision making are not necessarily the same as model rationales, the goal behind modeling human acceptability is often to build trust with a human user. This trust may or may not be warranted (Jacovi et al., 2021); future work would be well-suited to further investigate generated explanations for incorrect label predictions such as in Appendix C, which could mislead end users.
We thank Jena Hwang for helpful advice in designing our human user studies and Peter West for sharing GPT-3 prompting scripts. We thank other members of the Mosaic team at the Allen Institute for AI for valuable suggestions, as well as both our anonymous annotators and reviewers. This work was done while SW was an intern on Mosaic.
7 Ethics & Broader Impacts
All datasets used in this work are public, and we plan to release the machine-generated explanations and annotations we collected. We do not collect any personal information from our human participants.
Models that produce explanations in the means used in our experimental protocol (i.e., by conditioning on the gold labels) have the possibility to cause humans to place unwarranted trust in an AI system. This line of research is complementary to works investigating the faithfulness of model-generated free-text explanations (Hase et al., 2020; Wiegreffe et al., 2021). We demonstrate in Appendix C that GPT-3's explanations lack reliability because the model can explain answer choices that were not its prediction equally well. This may be due in part to the fact that decoding algorithms for generating predictions from language models are sub-optimal (e.g., Zhao et al., 2021; Holtzman et al., 2021) and GPT-3 may have factual knowledge stored in its parameters about other answer choices that allow it to provide reasonably acceptable explanations. Until this phenomenon is better understood, we do not condone using GPT-3-generated explanations in real-world deployment.
Lastly, our model of human acceptability is based on the aggregate judgements of participants from primarily Western, English-speaking countries working on crowdsourcing platforms. The subjective judgements of explanation acceptability may vary significantly among different population groups.
Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021. *Explanations for Common-senseQA: New Dataset and Models*. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics* and the 11th International Joint Conference on Natural Language Pro
cessing (Volume 1: Long Papers), pages 3050-3065, Online. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Faeze Brahman, Vered Shwartz, Rachel Rudinger, and Yejin Choi. 2021. Learning to rationalize for nonmonotonic reasoning with distant supervision. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, Online.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, Online. Curran Associates, Inc.
Oana-Maria Camburu, Tim Roktaschel, Thomas Lukasiewicz, and Phil Blunsom. 2018. e-snli: Natural language inference with natural language explanations. In Advances in Neural Information Processing Systems, volume 31, Montreal, Canada. Curran Associates, Inc.
Francisco Javier Chiyah Garcia, David A. Robb, Xingkun Liu, Atanas Laskov, Pedro Patron, and Helen Hastie. 2018. Explainable autonomy: A study of explanation styles for building clear mental models. In Proceedings of the 11th International Conference on Natural Language Generation, pages 99-108, Tilburg University, The Netherlands. Association for Computational Linguistics.
Miruna-Adriana Clinciu, Arash Eshghi, and Helen Hastie. 2021. A study of automatic metrics for the evaluation of natural language explanations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2376-2387, Online. Association for Computational Linguistics.
Upol Ehsan, Brent Harrison, Larry Chan, and Mark O Riedl. 2018. Rationalization: A neural machine translation approach to generating natural language explanations. In Proceedings of the 1st AAAI/ACM Conference on AI, Ethics, and Society (AIES), New Orleans, USA.
Upol Ehsan, Pradyumna Tambwekar, Larry Chan, Brent Harrison, and Mark O Riedl. 2019. Auto-
mated rationale generation: A technique for explainable ai and its effects on human perceptions. In Proceedings of the 24th International Conference on Intelligent User Interfaces (IUI), pages 263-274, Los Angeles, USA. ACM.
Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1161-1166, Hong Kong, China. Association for Computational Linguistics.
Alison Gopnik. 1998. Explanation as orgasm. *Minds and Machines*, 8(1):101-118.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
Peter Hase, Shiyue Zhang, Harry Xie, and Mohit Bansal. 2020. Leakage-adjusted simulatability: Can models generate non-trivial explanations of their behavior in natural language? In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 4351-4367, Online. Association for Computational Linguistics.
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. 2016. Generating visual explanations. In Proceedings of the 14th European Conference on Computer Vision (ECCV), Amsterdam, Netherlands.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In Proceedings of the 8th International Conference on Learning Representations (ICLR), Online.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038-7051, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alon Jacovi, Ana Marasovic, Tim Miller, and Yoav Goldberg. 2021. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai. In Proceedings of the 4th ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 624-635, Online.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, and Thomas Lukasiewicz. 2021. e-vil: A dataset and benchmark for natural language explanations in vision-language tasks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Online.
Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016. Examples are not enough, learn to criticize! criticism for interpretability. In Advances in Neural Information Processing Systems, volume 29, Barcelona, Spain. Curran Associates, Inc.
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John F. Canny, and Zeynep Akata. 2018. Textual Explanations for Self-Driving Vehicles. In Proceedings of the 15th European Conference on Computer Vision (ECCV), Munich, Germany.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, USA.
Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
Vivian Lai, Samuel Carton, and Chenhao Tan. 2020. Harnessing explanations to bridge ai and humans. In *Fair and Responsible AI Workshop at the CHI Conference on Human Factors in Computing Systems*, Online.
Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. 2022. Can language models learn from explanations in context? ArXiv preprint arXiv:2204.02329.
Veronica Latcinnik and Jonathan Berant. 2020. Explaining question answering models through text generation. ArXiv preprint arXiv:2004.05569.
David B Leake. 1991. Goal-based explanation evaluation. Cognitive Science, 15(4):509-545.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107-117, Austin, Texas. Association for Computational Linguistics.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumont, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid,