SlowGuess/NAACL_2022_OCR huggingface dataset:
$\mathbf{A T O M I C^{10x}}$  and 1000 from  $\mathrm{ATOMIC}_{20}^{20}$  (Table 4). We find Fleiss' kappa (Fleiss, 1971) of 40.8 indicating moderate agreement (Landis and Koch, 1977), and  $90.5\%$  accuracy agreement. We require workers meet an Amazon Mechanical Turk qualification for annotation quality based on past commonsense evaluations. We compensate workers  $\$0.17$  per task, which we estimate require 30 seconds. Further details and task template are in appendix  $\S A$ .
For the loose teacher, consider the top row of  $\mathrm{ATOMIC^{10x}}$  in Table 4 (other rows add the critic §4).  $\mathrm{ATOMIC^{10x}}$  exceeds  $\mathrm{ATOMIC_{20}^{20}}$  in scale, but is somewhat less acceptable by human raters-by roughly 8 percentage points. Yet, the larger scale of  $\mathrm{ATOMIC^{10x}}$  implies a significantly higher number of accurate examples. Increasing the proportion of these is the main objective of the critic (§4).
How do Knowledge Sources Compare? To understand the robustness of our approach, we assess other language models as the knowledge source (i.e. loose teacher): GPT-J (Wang and Komatsuzaki, 2021) and T5-11B adapted for language modelling (Lester et al., 2021). We substitute both for GPT-3 as in §3.2,3.3, generating a small-scale corpus to evaluate. We conduct human evaluation on 1000 examples as above (Table 4). Both models attain roughly  $72\%$  accuracy, 6 points below GPT-3 (78.5). This suggests strong potential, but higher quality from GPT-3. We explore this further in Appendix B.
${}^{9}$  Size of ATOMIC  ${}_{20}^{20}$  is given as the number of comparable datapoints,i.e. those with the same relations as ATOMIC  ${}^{{10}\mathrm{x}}$  .
this ensures at least 1000 after filtering by the critic  $\S 4$
4 Making the Teacher More Critical
Symbolic knowledge distillation requires a strong teacher model to maximize the quality of the generated knowledge graph and resulting student model (§5). While the loose teacher (GPT-3 alone) results in a viable commonsense knowledge graph, evaluation shows this isn't a perfect commonsense teacher. Thus, we multiply in a critic model, to filter lower-quality knowledge, correcting the teacher (§2.1). With modest supervision (a small-scale human evaluation) we train a classifier to predict and discriminate unacceptable examples. We multiply this with the loose teacher §3, creating a critical teacher product of experts. In practice this means filtering  $\mathrm{ATOMIC^{10x}}$  to create new corpora that are higher quality, yet still larger scale than human-authored  $\mathrm{ATOMIC_{20}^{20}}$ .
Training a knowledge critic We gather a training set of correct vs. incorrect human judgments on a randomly-sampled set of 10K entries of ATOMIC $^{10x}$ , as in §3.4 but with one annotation per example. We take a (random) train/dev/test split of  $8k/1k/1k$ . While this step requires human annotation, humans take on the role of high-level supervisors here—critiquing a small number of generations rather than authoring the entire knowledge graph as in previous work. Indeed, the cost/complexity of this step is similar to a typical human evaluation, making it far cheaper/easier than eliciting human-authored knowledge in past work.
We train binary classifiers (critics) for human acceptability using RoBERTa-Large (Liu et al., 2019). We find pretraining on MNLI results in the best model in terms of precision and recall, and we suggest this technique for future studies. We give more detail in Appendix C, including baselines. Our best model vastly improves the accuracy of ATOMIC $^{10x}$  (Table 4), demonstrating that a small amount of human supervision can consistently help to correct GPT-3's mistakes.
Size-accuracy trade-off Using our critic to filter knowledge results in a natural trade-off between size and accuracy. We test several cutoffs for  $\mathrm{ATOMIC^{10x}}$ , i.e. confidence at which the critic rejects examples. We report human-measured accuracy (Accept/Reject column Table 4) following §3.4. We compare the loose teacher (unfiltered) to critical teachers. Discarding  $20\%$  of instances that the critic judges as least acceptable (reducing corpus size from 6.5M to
Random	Inf	Event	EMAP	Full
AP	79.3	81.9	86.2	87.1	94.0
Table 5: Average Precision for ablated critic models. The critic not only filters awkward phrasings which can be identified by either the event (Event) or inference (Inf) in isolation (EMAP only identifies these), but also logical misalignments, which require modeling interactions between event/inference, i.e. the full critic (Full).
5.1M), ATOMIC $^{10x}$ ’s accuracy rises  $78.5 \rightarrow 88.4$ ; human-authored ATOMIC $^{20}_{20}$  contains 600K entries at  $86.8\%$  accuracy. Reducing to total size to 2.5M examples (38% of full size), we attain  $96.4\%$  accuracy, nearly 10 points above ATOMIC $^{20}_{20}$  while still 4X larger.
What gets filtered out? We qualitatively identify two types of filtered triples: 1) logical misalignments, events/inferences joined in an inconsistent manner. Recognizing these requires understanding events-inference interactions, e.g.,  $X$  cannot find his shirt as a result  $X$  is wearing a shirt; 2) awkward phrasings, in which events/inferences are individually incoherent e.g. PersonX has a fire in the bath—resulting triples are invalid as the event is implausible.
To understand what is filtered, we ablate the critic (Table 5): our full model is compared to a random predictor, event-only model, and inference-only model. We also compare to an EMAP (Hessel and Lee, 2020) version, i.e. an ensemble of event and inference-only, without interactions between event/inference (needed for logical misalignments).
We find GPT-3 produces both independent awkwardly-phrased events/inferences (filtered by X-only models) and logical misalignments. The classifier, trained on validated knowledge triples, helps in both cases. The EMAP of our full model (identifies only awkward phrasings) achieves  $87\%$  AP, and our full model (which additionally identifies logical misalignments) improves to  $94\%$  AP.
Does filtering hurt diversity? One concern is that the critic may keep only similar "safe" examples, lacking novelty. We repeat our diversity analysis (§3.4) for critical corpora (Table 4, "Size (div)", higher=better). As we filter, we surprisingly observe proportionally more diverse examples: full ATOMIC $^{10x}$  has a diverse subset  $68\%$  of its size; rising to  $80\%$  with the most extreme filtering. One possibility is that GPT-3 gravitates towards com
CKG Completion Model	Train Corpus Acc	Accept	Reject	N/A
GPT2-XL zero-shot	-	45.1	50.3	4.6
GPT-3	-	73.3	24.1	2.6
COMET20	86.8	81.5	16.3	2.2
COMETDIS	78.5	78.4	19.2	2.4
+criticlow	91.5	82.9	14.9	2.2
+critichigh	96.4	87.5	10.2	2.3
Table 6: Model performance on knowledge base completion, measured by human judgement. Inferences are generated on held-out events from  $\mathrm{ATOMIC}_{20}^{20}$ . Models besides GPT-3 use GPT-2 XL architecture. COMET $_{\text{TIL}}^{\text{DIS}}$  with a strong critic (+critichigh) achieves the highest acceptance rate overall-87.5.
mon sentence structures for inconsistent knowledge. These would be recognizable to the critic, and removing them would increase both quality and diversity. This surprising result warrants further study.
5 Corpus-to-Machine: Distillation
The final step of symbolic knowledge distillation trains a compact model on the generated natural language knowledge graph. Our base model is GPT2-XL trained on all of ATOMIC $^{10x}$ : we denote this model by  $\mathrm{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$ . We additionally train the model on critical versions of ATOMIC $^{10x}$ -crit $_{\mathrm{low}}$  denotes training on the corpus achieving  $91.5\%$  accuracy, and  $\mathrm{crit}_{\mathrm{high}}$  on the  $96.4\%$  accuracy corpus. Models are trained for 1 epoch, with default parameters using the Huggingface Transformers library (Wolf et al., 2019).
5.1 Evaluating a Symbolically Distilled Model
Evaluation follows past work (Hwang et al., 2021; Bosselut et al., 2019; Sap et al., 2019) testing the ability of models to do knowledge base completion, i.e. generating inferences for test events, specifically from the  $\mathrm{ATOMIC}_{20}^{20}$  test set. We use human evaluation $^{10}$  following Section 3.4, on 1000 inputs (event + relation), with results in Table 6. We compare to the GPT2-XL-based  $\mathrm{COMET}_{20}^{20}$  model trained on human-generated  $\mathrm{ATOMIC}_{20}^{20}$ , and GPT-3 using the same generation method as §3-in effect, comparing the student  $\mathrm{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$  to the loose teacher GPT-3. We omit the critical teacher (GPT-3 + critic), which is not assured to produce an in
ference for each input, as the critic may reject all tails for some inputs. We also compare to zero-shot GPT2-XL (Radford et al., 2019) using the same methodology (Table 6).
How does  $\mathrm{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$  compare to GPT-3? In knowledge distillation, the student model often deteriorates in performance (Hinton et al., 2015; Kim and Rush, 2016) compared to its teacher. Comparing our base teacher-GPT-3-to the simplest version of  $\mathrm{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$  (top-row  $\mathrm{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$  of Table 6) surprisingly shows the student surpasses GPT-3, the model that generates its training data<sup>11</sup>. We posit that the superior performance of  $\mathrm{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$  may have to do with mistakes of GPT-3 being filtered by verbalization and training of GPT-2, and possibly the focus of  $\mathrm{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$  on one commonsense domain while GPT-3 covers a more general domain. We leave further study of this effect for future work.
How does  $\mathrm{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$  compare to human knowledge? While  $\mathrm{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$  without the critic is slightly outperformed by  $\mathrm{COMET}_{20}^{20}$  in terms of accuracy, this reverses with the critic. For both cutoffs tested,  $\mathrm{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$  surpasses  $\mathrm{COMET}_{20}^{20}$ , with more filtering resulting in a wider gap.
Usefulness of  $\mathbf{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$  For on-demand inference, where a single high quality inference for some input event/relation is required,  $\mathbf{COMET}_{\mathrm{TIL}}^{\mathrm{DIS}}$  is the best available model: the most performant version surpasses  $\mathbf{COMET}_{20}^{20}$  by 5 points and GPT-3 by over 10. The critical teacher (GPT-3 + critic) yields a more accurate corpus, but may filter all inferences for an input, giving no output.
Limits and Future Work The success of symbolic knowledge distillation is a first step—demonstrating superior performance to human authoring on the commonsense relations tested here. No aspect of our approach is specific to these relations, yet further work is needed to explore the feasibility of generation for other aspects of commonsense and knowledge, beyond these relations, to concepts like physical or temporal commonsense.
Commonsense Knowledge Graphs (CKG) CKGs provide knowledge for commonsense reasoning. Some are manually constructed, e.g.
11The slight difference in acceptability for GPT-3 from Table 4 is likely due to variance in raters between rounds of evaluation, and a different distribution of events-Table 4 uses generated events while Table 6 uses events from ATOMIC20.
<sup>10</sup>We find Fleiss' kappa (Fleiss, 1971) of 47.1 for acceptance, indicating moderate agreement. (Landis and Koch, 1977), and accuracy agreement of  $88.7\%$ .
ATOMIC (Sap et al., 2019; Hwang et al., 2021). ConceptNet (Speer et al., 2017) contains taxonomy and physical commonsense, authored by humans or compiled from such sources. Some CKGs are automatically constructed: TransOMCS (Zhang et al., 2020a) extracts 18.48M tuples from syntactic parses and CausalBank (Li et al., 2020) extracts 314M cause-effect pairs by pattern-matching. In contrast, we generate commonsense.
Extracting Knowledge from LMs Past work uses models for automatic knowledge graph completion (Bosselut et al., 2019; Hwang et al., 2021; Li et al., 2020). Yet, models are trained on existing resources; ATOMIC $^{10x}$  is generated without these. Other works mine factual/common sense knowledge directly from off-the-shelf LMs (Petroni et al., 2019; Davison et al., 2019; Xiong et al., 2020), but not resulting in the quality at scale of ATOMIC $^{10x}$ .
Knowledge Distillation Other works use knowledge distillation (Hinton et al., 2015) for generation. (Sanh et al., 2019) follow a label smoothing formulation, while Kim and Rush (2016) follow a similar formulation to us (§2.1), use the mode of the teacher distribution rather than sampling. Our work is unique in distilling specific information (commonsense) from a general language model.
Data Generation While manual dataset creation is expensive and complex (Schwartz et al., 2017; Agrawal et al., 2018; Tsuchiya, 2018; Bras et al., 2020), crowdsourcing is the most popular method for goal-oriented, high quality/coverage datasets.
Past automatic data mainly use extractive approaches, e.g. syntactic parsing (Zhang et al., 2020a) or pattern matching (Li et al., 2020) from unstructured text (Lehmann et al., 2015; Buck et al., 2014). These scale, but are noisy and limited in format-ATOMIC knowledge will not appear simply in natural text. Some works explore automatic data synthesis/expansion by finetuning LMs on existing labeled data (Anaby-Tavor et al., 2020; Papanikolaou and Pierleoni, 2020; Kumar et al., 2020; Yang et al., 2020), but are limited by data quality.
We introduce symbolic knowledge distillation, a machine-to-corpus-to-machine pipeline for commonsense that does not require human-authored knowledge—instead, using machine generation. Knowledge is transferred from a large, general model to a compact commonsense model, through
a commonsense corpus-yielding a commonsense knowledge graph and model. Our resulting symbolic knowledge graph has greater scale, diversity, and quality than human authoring. symbolic knowledge distillation offers an alternative to human-authored knowledge in commonsense research.
This work was funded in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) (funding reference number 401233309), DARPA MCS program through NIWC Pacific (N66001-19-2-4031), and the Allen Institute for AI.
One aspect of our work with the potential for ethical pitfalls is large-scale generation from pretrained language models, in constructing  $\mathrm{ATOMIC^{10x}}$ . Recent work (Bender et al., 2021) has highlighted the risks of models trained on massive text resources, as GPT-3 (Brown et al., 2020) is, which we use for generation. Indeed, open generations from pretrained language models can often contain harmful, biased, or offensive aspects. We argue here that this risk is largely mitigated in our work, mainly due to the narrow and constrained nature of our generations. The goal of our work is characterising simple and generic anonymous situations, specifically in terms of commonsense causes and effects. We ensure generations are focused on these topics through careful prompting, which we found to be quite effective at keeping these generations on-topic. As such, the potential for harmful generation is very low; indeed, in a manual inspection of 100 generated examples, we found none that were significant harmful, besides one that contained adult content.
A related concern is the potential for large models and training sets to make automated oppression or exploitation possible, for instance in surveillance or generating fake news. As above, we argue that the generic, commonsense nature of our data and models makes this concern less relevant here. Our data does not contain any information directly related to these harmful domains (e.g. social media or fake news generation). While our data may assist machines in understanding basic situations, this is unlikely to be useful for harmful models given the simplicity of our data and still-flawed commonsense capabilities of even the most advanced
Finally, we note that we ensure fair and generous compensation for all human evaluators we hire through Amazon Mechanical Turk. Based on our estimates of time required per task, we ensure that the effective pay rate is at least $15 per hour.
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don't just assume; look and answer: Overcoming priors for visual question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4971-4980.
Prithviraj Ammanabrolu, Wesley Cheung, William Broniec, and Mark O. Riedl. 2021a. Automated storytelling via causal, commonsense plot ordering. In AAAI.
Prithviraj Ammanabrolu, Jack Urbanek, Margaret Li, Arthur D. Szlam, Tim Rocktaschel, and Jason Weston. 2021b. How to motivate your dragon: Teaching goal-driven agents to speak and act in fantasy worlds. In *NAACL*.
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! Proceedings of the AAAI Conference on Artificial Intelligence, 34:7383-7390.
Forough Arabshahi, Jennifer Lee, Antoine Bosselut, Yejin Choi, and Tom. Mitchell. 2021. Conversational multi-hop reasoning with neural commonsense knowledge and symbolic logic rules. In EMNLP.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
Sumithra Bhakthavatsalam, Chloe Anastasiades, and Peter E. Clark. 2020. Genericsb: A knowledge base of generic statements. ArXiv, abs/2005.00660.
Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD Conference.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, A. Celikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for automatic knowledge graph construction. In ACL.
Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew E. Peters, Ashish
Sabharwal, and Yejin Choi. 2020. Adversarial filters of dataset biases. In ICML.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Christian Buck, Kenneth Heafield, and Bas Van Ooyen. 2014. N-gram counts and language models from the common crawl. In LREC, volume 2, page 4. Citeseer.
Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. 2020. R^3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In ACL.
Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, and Nanyun Peng. 2021. MERMAID: Metaphor generation with symbolism and discriminative decoding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4250-4261, Online. Association for Computational Linguistics.
Ernest Davis and Gary Marcus. 2017. Causal generative models are just a start. Behavioral and Brain Sciences, 40.
Joe Davison, Joshua Feldman, and Alexander M Rush. 2019. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173-1178.
Oren Etzioni, Anthony Fader, Janara Christensen, Stephen Soderland, et al. 2011. Open information extraction: The second generation. In Twenty-Second International Joint Conference on Artificial Intelligence.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
Jack Hessel and Lillian Lee. 2020. Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think! In EMNLP.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.
Geoffrey E Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. AAAI.
William R. Kearns, Neha Kaura, Myra Divina, Cuong Viet Vo, Dong Si, Teresa M. Ward, and Weichao Yuwen. 2020. A wizard-of-oz interface and persona-based methodology for collecting health counseling dialog. Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In EMNLP.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained transformer models. In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems, pages 18-26, Suzhou, China. Association for Computational Linguistics.
J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159-174.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, D. Kontokostas, Pablo N. Mendes, Sebastian Hellmann, M. Morsey, Patrick van Kleef, S. Auer, and C. Bizer. 2015. Dbpedia - a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6:167-195.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In EMNLP.
Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, and Benjamin Van Durme. 2020. Guided generation of cause and effect. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. 2021. Provable limitations of acquiring meaning from ungrounded form: What will future language models understand? Transactions of the Association for Computational Linguistics, 9:1047-1060.