Venue: NAACL 2022

# Symbolic Knowledge Distillation:

from General Language Models to Commonsense Models

 Peter West\({}^{\dagger\ddagger}\)   Chandra Bhagavatula\({}^{\ddagger}\)    Jack Hessel\({}^{\ddagger}\)  Jena D. Hwang\({}^{\ddagger}\)

**Liwei Jiang\({}^{\ddagger}\)**   Ronan Le Bras\({}^{\ddagger}\)   Ximing Lu\({}^{\dagger\ddagger}\)   Sean Welleck\({}^{\dagger\ddagger}\)   Yejin Choi i\({}^{\ddagger*}\)

\({}^{\dagger}\)Paul G. Allen School of Computer Science & Engineering, University of Washington

\({}^{\ddagger}\)Allen Institute for Artificial Intelligence

###### Abstract

The common practice for training commonsense models has gone _from-human-to-corpus-to-machine-_ humans author commonsense knowledge graphs in order to train commonsense models. In this work, we investigate an alternative, _from-machine-to-corpus-to-machine-_ general language models author these commonsense knowledge graphs to train commonsense models.

Our study leads to a new framework, **Symbolic Knowledge Distillation**. As with prior art in Knowledge Distillation [20], our approach uses larger models to teach smaller models. A key difference is that we distill knowledge symbolically-as text-in addition to the resulting neural model. We distill only one aspect-the commonsense of a general language model teacher, allowing the student to be a different type of model, a commonsense model. Altogether, we show that careful prompt engineering and a separately trained critic model allow us to selectively distill high-quality causal commonsense from GPT-3, a general language model.

Empirical results demonstrate that, for the first time, a _human-authored_ commonsense knowledge graph is surpassed by our _automatically distilled_ variant in all three criteria: quantity, quality, and diversity. In addition, it results in a neural commonsense model that surpasses the teacher model's commonsense capabilities despite its 100x smaller size. We apply this to the Atomic resource, and will share our new symbolic knowledge graph and commonsense models1.

Footnote 1: We will share this following the anonymity period. We have permission from OpenAI to release GPT-3 generations

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4602 - 4625

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Prior works have suggested that pre-trained language models possess limited understanding of commonsense knowledge [8, 9, 7] despite otherwise stellar performance on leaderboards. As a result, symbolic commonsense knowledge graphs [23, 24, 25] and corresponding neural representations [22, 26] have supplemented past models with commonsense capabilities. This has enabled diverse downstream applications, including interactive learning through a conversational interface [1], persona- and affect-aware conversation models [11], figurative language understanding [15, 20], story telling [16] and fantasy games [17].

The common practice for commonsense knowledge graph construction sees humans spell out as many pieces of knowledge as possible. This pipeline goes _from-human-to-corpus-to-machine_, with commonsense models trained from human

Figure 1: **Symbolic knowledge distillation extracts the commonsense from the large, general language model GPT-3, into 2 forms: a large commonsense knowledge graph \(\textbf{Atomic}^{\text{tos}}\), and a compact commonsense model \(\textbf{Comet}^{\text{tos}}_{\text{ttl}}\). The quality of this knowledge can be controlled and improved by adding a **critic** model, making GPT-3 a stronger teacher.**

authored knowledge graphs. Yet, high-quality, human-authored knowledge is expensive to scale, limiting coverage; this motivates an alternative: _from-machine-to-corpus-to-machine_. Prior efforts toward automatic commonsense knowledge graphs have resulted in considerably lower quality than human-written data Hwang et al. (2021); Zhang et al. (2020b), which in turn leads to less reliable neural models Hwang et al. (2021). Broad literature consistently shows machine-authored knowledge graphs underperform human-authored graphs Etzioni et al. (2011); Mitchell et al. (2015); Bollacker et al. (2008).

In this work, we propose **Symbolic knowledge distillation**, a new conceptual framework towards high-quality automatic knowledge graphs for commonsense, leveraging state-of-the-art models and novel methodology. Most prior art for automatic knowledge graph construction extracts knowledge from raw text Bhakhtavasalam et al. (2020); Zhang et al. (2020a); Zhou et al. (2020); Zhang et al. (2020b); Li et al. (2020). In contrast, our approach is motivated by knowledge distillation Hinton et al. (2015) wherein a larger teacher model transfers knowledge to a compact student model (SS2.1). Our method differs from prior knowledge distillation in key ways: we distill a symbolic knowledge graph (i.e., generated text) in addition to a neural model, and we distill only a selective aspect of the teacher model. This selectively allows the student model to be of a different type (commonsense model), compared to the teacher (general language model), enriching the scope of distillation. An added benefit is that knowledge distilled as text is human readable: it can be understood and evaluated.

A general language model-GPT-3 in our case-is an imperfect commonsense teacher on its own, and the ability to evaluate distilled knowledge is useful in improving it. We empirically demonstrate that, by training a separate critic model to judge symbolic generation quality, a more precise teacher can be defined. Knowledge from this critical teacher is higher quality-even exceeding human-authored knowledge. Yet even before training a critic, our study makes the unexpected finding that the student model surpasses the commonsense of GPT-3, our knowledge source.

To test symbolic knowledge distillation against the _human-to-corpus-to-machine_ paradigm, we compare with Atomic\({}^{20}_{20}\)Hwang et al. (2021), which is a human-authored commonsense knowledge graph. We find that Atomic\({}^{\textbf{10x}}\), our machine-generated corpus, exceeds the human generated corpus in _scale, accuracy,_ and _diversity_ with respect to 7 commonsense inference types that we focus on in this study. The resulting commonsense model, \(\textbf{Comet}^{\textsc{DIS}}_{\textsc{till}}\), not only surpasses the human-trained equivalent Comet\({}^{20}_{20}\), but is also smaller, more efficient, and produces commonsense at a higher accuracy than its own teacher-GPT-3.

Symbolic knowledge distillation offers a promising new role for general language models, as commonsense knowledge sources, and humans, as small-scale evaluators to train critic models rather than authors of commonsense knowledge. Our work demonstrates that humans and LMs can be effective collaborators for curating commonsense knowledge graphs and training efficient and performant commonsense models.

## 2 Overview and Key Findings

Throughout our work, we describe the _machine-to-corpus-to-machine_ methodology of symbolic knowledge distillation. We first go _machine-to-corpus_ (SS3), by decoding from GPT-3, then improve our knowledge with a specialized critic model (SS4), and finally distill this knowledge into an efficient commonsense model (SS5), going _corpus-to-machine_. Throughout this process, we evaluate against a human knowledge source, comparing our automatic knowledge graph Atomic\({}^{\textbf{10k}}\) and commonsense model Comet\({}^{\textsc{dis}}_{\textsc{till}}\) to the human-autured Atomic\({}^{20}_{20}\) and resulting model Comet\({}^{20}_{20}\)Hwang et al. (2021).

### Symbolic Knowledge Distillation

Our proposed methodology parallels knowledge distillation Hinton et al. (2015), a method for compressing a large or complicated teacher distribution \(P_{t}\) into a smaller/simpler student distribution \(P_{s}\). Key to knowledge distillation2 is the notion of minimizing the cross-entropy between \(P_{t}\) and \(P_{s}\):

Footnote 2: In its simplest case, with temperature set to 1.0

\[H(P_{t},P_{s})=-\sum_{y\in Y}P_{t}(y)\log P_{s}(y) \tag{1}\]

Knowledge is transferred to the student by encouraging it to match teacher predictions. Hinton et al. (2015) apply this to conditional classification: for 

each training input, \(P_{t}\) and \(P_{s}\) are model predictions over label set \(Y\). Typically \(Y\) is a tractable set, over which this sum can reasonably be calculated.

For distilling the knowledge of generative models, we can think of an unconditional language model (LM e.g. GPT-3) as \(P_{t}\). This makes \(Y\) the set of all strings, over which LMs define probability. Unfortunately \(Y\) is an exponential set, intractable to sum over in Eq 1. Kim and Rush (2016) address this problem by simply taking the mode of \(P_{t}\) over \(Y\), truncating most of the teacher distribution to the most likely sequence and discarding information.

Instead, we consider a sampling-based interpretation of the same objective:

\[H(P_{t},P_{s})=\mathop{\mathbb{E}}_{y\sim P_{t}(y)}[-\log P_{s}(y)] \tag{2}\]

which exactly equals the cross-entropy of Eq 1, at the limit under pure sampling from \(P_{t}\).3

Footnote 3: A useful consequence of this framing is that access to the full model distribution is not required. Our experiments (§3) use GPT-3, for which the distribution is **not available**, thus our method is applicable while knowledge distillation is not.

Yet distilling _all knowledge_ from the teacher may not be desirable--our work is specifically focused on distlling commonsense knowledge from GPT-3. The ideal teacher \(P_{t}\) is a commonsense expert, but GPT-3 can approximate such a teacher, off-the-shelf, via prompting. This ability to select information is one explicit benefit of the sampling-based interpretation of Eq 2: while Eq 1 uses continuous logits over existing data, sampling gives discrete control over transferred information, by selecting which samples are elicited and used. For the general language model GPT-3, We encourage domain/quality with prompting, and sample truncation Holtzman et al. (2020). We call this the _loose teacher_\(P^{L}_{t}\)-knowledge is generated and transferred from GPT-3, but without critical assessment of correctness (SS3).

In fact, sampling knowledge in Eq 2 offers even more control, as generations can be individually interpreted and judged. Given an indicator function \(A(x)\) for which knowledge \(x\) is _correct_, we can define a stronger teacher model. Using a Product of Experts Hinton (2002) between the loose teacher \(P^{L}_{t}\) and and the critic \(A(x)\), we define a _critical teacher_:

\[P_{t}(x)\propto P^{L}_{t}(x|p)\cdot A(x) \tag{3}\]

In practice, \(A(x)\) is a textual classifier learned on human judgements, 1 for knowledge predicted to be correct and 0 otherwise. Thus, the critic gives control over the correctness and confidence of the knowledge that is transferred (SS4).

### Key Findings

Applying symbolic knowledge distillation in practice results in promising and surprising findings:

**1. Learning symbolic knowledge from language models can be framed as a symbolic extension to knowledge distillation.** In SS2.1, we describe learning commonsense as a symbolic extension to knowledge distillation, with GPT-3 a knowledge source. We elaborate on this process with positive results in SS3,4, and 5.

**2. Symbolic knowledge distillation constructs a high quality knowledge graph at scale.** Our method naturally yields a machine-generated commonsense knowledge graph, which can achieve impressive quality (SS4), beyond that of human-authored data. An effective critic which filters incorrect generated knowledge is key.

**3. A critical teacher results in a higher quality student.** In SS4, we show that making the teacher more critical results in higher quality knowledge, even as it reduces the scale of knowledge transferred. This demonstrates that _quality_ matters, not

Figure 2: Example **automatically generated Atomic triples from our Atomic\({}^{\mathbf{108}}\) commonsense knowledge graph. Each example includes a generated event, relation** (with natural language interpretation), and generated inference.

 

just _quantity_, as higher quality knowledge results in a higher quality commonsense model in SS5 despite smaller scale data.

Footnote 3: [https://github.com/en-tran/](https://github.com/en-tran/)

4. Critical teacher or not, a student can outperform the knowledge source.In SS5, we show the unexpected result that all student models exceed the quality of GPT-3, the knowledge source.

5. Machines can win over humans for automatic knowledge graph construction.In SS4 and SS5, we show that machine generated knowledge and the resulting commonsense model can outperform their equivalents that use a human knowledge source. Our symbolic knowledge exceeds humans at scale, quality, and diversity. The resulting commonsense model achieves the most accurate commonsense KG completions.

## 3 Machine-to-Corpus Verbalization

Symbolic knowledge distillation begins by going _machine-to-corpus_, i.e. generating many commonsense facts, which results in a commonsense knowledge graph. SS2.1 frames this as sampling to estimate the knowledge distillation objective-a student commonsense model learns from the generations of a teacher (GPT-3).

We start with a _loose teacher_, transferring knowledge by prompted generation with truncated sampling alone-this is in contrast to the _critical teacher_ (SS4) which explicitly judges and filters the generated samples. The loose teacher uses few-shot prompting as in Brown et al. (2020). We use a few-shot template:

```
<TASK-PROMPT> <EX\({}_{1}\)--IMP><EX\({}_{1}\)--OUT> ... <EX\({}_{N-1}\)--INP><EX\({}_{N-1}\)--OUT> <EX\({}_{N}\)--INP>
```

where <EX\({}_{i}\)--INP>/<EX\({}_{i}\)--OUT> are human-authored, natural language Atomic entries, and <TASK-PROMPT> is a description of the problem. Given such a prompt, GPT-3 generates the _missing piece_, output <EX\({}_{N}\)--OUT> for input <EX\({}_{N}\)--INP>, following the pattern of earlier examples (1 to N-1). We find important aspects for producing high-quality commonsense knowledge:

* Examples should be numbered. e.g. <EX\({}_{5}\)--INP> might begin with "5)" to indicate it is the 5th example.
* The format of <EX\({}_{i}\)--INP> and <EX\({}_{i}\)--OUT> should linguistically imply the relationship between them. See below for examples.
* <TASK-PROMPT> can be used to give extra specification to complicated problems.

### Data: Atomic

We demonstrate symbolic knowledge distillation on the Atomic _if-then_ resource (Sap et al., 2019). This follows an event-relation-inference (triple) format. The corpus links _events_ (e.g. _X attacks Y_) to relations, e.g. **HinderedBy** which describes what might hinder an event. For a relation/event, the goal is to generate a resulting inference, e.g. _X attacks Y_**HinderedBy**_X is restrained_.

Of the 23 relations from the most recent version-Atomic\({}^{20}_{20}\)-we limit our investigation to 7 relations that correspond to _causal_commonsense knowledge: **xAttr** (how X is perceived after _event_), **xRecat** (how X reacts in response to _event_), **xEffect** (what X does after _event_), **xIntent** (X's intent in _event_), **xWant** (what X wants after _event_), **xNeed** (what X needed for _event_ to take place) and **HinderedBy**. We describe how **verbalization** is applied to Atomic data in 2 steps: generating underlying events (heads), then full examples (inference given event).

### Event Generation

Events are context-free premises in Atomic involving PersonX (and sometimes a second PersonY) in various scenarios. These events form heads in knowledge graph triples. We generate events by filling in the elements of our template:

```
1.Event:X overcomesevilwithgood  2.Event:XdoesnotlearnfromY  ...
```

The format is simple, as events are generated _unconditionally_. We use 100 high-quality events from the Atomic\({}^{20}_{20}\) corpus for our prompt, selected to avoid grammatical or logical errors, and minimize semantic overlap. We randomly sample 10 of these seed events for each generation batch, resulting in randomized prompts. We use nucleus sampling (\(p=0.9\)) (Holtzman et al., 2020), and presence/frequency penalties of 0.5 from the GPT-3 interface. We generate 165K unique events using the 175B-parameter Davinci model4 from Brown 

et al. (2020) (human-authored atomic\({}^{20}_{20}\) contains only 6.2K events).

### Inference Generation

Generating Atomic inferences requires reasoning about events and relations together. We design verbalization templates fo reach relation, with iterative design and small-scale verification by the authors5 e.g. we prompt the **xNeed** relation as follows:

Footnote 5: See Appendix D for full prompts.

What needs to be true for this event to take place?  ...  Event <i>: X goes jogging Prerequisites: For this to happen, X needed to wear running shoes  ...  Event <N>: X looks at flowers  Prerequisites: For this to happen, The language of this template implies the relationship task, both "Prerequisites:" and beginning with "for this to happen" suggest the **xNeed** relation. As well, we include an xNeed-specific <TASK-PROMPT>. We use 10 few-shot examples for each prompt.6

Footnote 6: We also replace anonymous names (“X”) with sampled generic names as this improved quality, See Appendix D. Once generation is complete, we substitute in generic markers (“X”) for the final dataset.

For each event/relation (165K X 7) we generate 10 inferences with the Curie GPT-3 model7 and earlier hyperparameters. Removing duplicate and degenerate (e.g. fewer than 3 characters) generations yields 6.46M Atomic-style data triples (examples in Figure 2). We call this Atomic**10x**, as it contains an order of magnitude more triples than Atomic\({}^{20}_{20}\) for the 7 relations we study.

Footnote 7: for the largest, Davinci, 12M generations is computationally/monetarily intractable.

### Evaluating a Generated Commonsense Knowledge Graph

Machine generation enables a large scale of unique generations at a much lower cost than human-authored knowledge (Table 1), but what kind of examples are produced by GPT-3, and how does it differ from knowledge produced by humans? In this section, we conduct an in-depth analysis to answer these questions.

#### Lexical Differences: Diversity and Uniqueness

Recent work finds that machine generations can be repetitive and lack diversity Welleck et al. (2020); Holtzman et al. (2020); one way generated knowledge may differ from human-authored is less creative word choice, diversity, or more repetition.

To test this, we begin with lexical diversity (i.e. unique words used, Table 2). While there is variation by relation, the diversity of Atomic**10x** actually exceeds Atomic\({}^{20}_{20}\) here, 5.2M unique words to 1.5M. In addition, it contains significantly more strictly unique generated inferences (Table 2, unique tails).

BLEU Soft Uniqueness.Exact match (above) fails to capture the notion of _similar_ text. Following the intuition of self-BLEU Zhu et al. (2018), we define _soft uniqueness_ to describe diversity of generations in a corpus. An inference \(x\) is softly-unique if:

\[BLEU_{2}(C,x)<0.5\]

where \(C\) is the set of inferences for a given input (in our case, event + relation), and 0.5 is an empirical threshold. To find soft-uniqueness of a corpus, we iteratively remove examples until all are softly unique, i.e. low mutual lexical overlap; higher diversity means more such examples (thus a larger softly unique corpus is preferable). Softly-unique corpus sizes are given in Table 4 ("Size (div)"). Atomic**10x** has a smaller _fraction_ of softly-unique examples than Atomic\({}^{20}_{20}\), yet it contains many more such examples. Atomic**10x** contains 4.38M such examples (full size 6.5M) vs. Atomic\({}^{20}_{20}\), which has 560K (full size 600K).

\begin{table}
\begin{tabular}{l r r} \hline \hline
**Relation** & Atomic\({}^{20}_{20}\) & Atomic**10x** \\ \hline HinderedBy & 77,616 & **1,028,092** \\ xNeed & 100,995 & **760,232** \\ xWant & 109,098 & **730,223** \\ xIntent & 54,839 & **965,921** \\ xReact & 62,424 & **1,033,123** \\ xAttr & 113,096 & **884,348** \\ xEffect & 90,868 & **1,054,391** \\ \hline
**Total Count** & 608,936 & **6,456,300** \\
**Est Total Cost** & \textasciold{$~{}\sim$}540,000 & \textasciold{$~{}\sim$}**56,000** \\
**Est Cost Per Triple** & \textasciold{$~{}\sim$}50.06 & \textasciold{$~{}\sim$}**50.001** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Number of unique triples with the given relation, \(|(\cdot,\texttt{relation},\cdot)|\). The estimated cost for Atomic**10x** comes at a fraction of a conservative estimation for Atomic\({}^{20}_{20}\) crowdsourcing costs.

 

Model-based Diversity Measurement.Lexical notions of diversity reward differences in surface form, which may not always reflect diversity of _information_, only format. Thus, we next study information-theoretic measures for diversity. Intuitively, diverse information should be less predictable, or higher entropy. With GPT-2 XL models finetuned on Atomic\({}^{20}_{20}\) and Atomic\({}^{108}\) (SS5) we estimate **entropy**-roughly, how difficult it is for a model to capture the corpus information (Table3). This is 4 times higher for Atomic\({}^{108}\), suggesting more content from a modeling perspective. We also estimate **cross-entropy**-how well a model trained on one corpus describes the other. From Atomic\({}^{108}\) to atomic\({}^{20}_{20}\), this is 9.31, only 2 points higher than its entropy suggesting Atomic\({}^{20}_{20}\) is describable with information from Atomic\({}^{108}\). In reverse, this is 41.48 suggesting much of Atomic\({}^{108}\) is not captured by Atomic\({}^{20}_{20}\)-Atomic\({}^{108}\) is surprising given only information from Atomic\({}^{20}_{20}\).

Human Evaluation of Quality.Perhaps most importantly, we study the _quality_ of knowledge in each corpus. We conduct human evaluation with Amazon Mechanical Turk. 3 annotators rate each triple resulting in "accepted", "rejected" or "no judgement". We evaluate 3000 examples8 from Atomic\({}^{108}\), and 1000 from Atomic\({}^{20}_{20}\) (Table4). We find Fleiss' kappa [7] of 40.8 indicating moderate agreement [6], and 90.5% accuracy agreement. We require workers meet an Amazon Mechanical Turk qualification for annotation quality based on past commonsense evaluations. We compensate workers $0.17 per task, which we estimate require 30 seconds. Further details and task template are in appendix SSA.

Footnote 8: this ensures at least 1000 after filtering by the critic §4)

For the _loose teacher_, consider the top row of Atomic\({}^{108}\) in Table4 (other rows add the critic SS4). Atomic\({}^{108}\) exceeds Atomic\({}^{20}_{20}\) in scale, but is somewhat less acceptable by human raters-by roughly 8 percentage points. Yet, the larger scale of Atomic\({}^{108}\) implies a significantly higher _number_ of accurate examples. Increasing the proportion of these is the main objective of the critic (SS4).

How do Knowledge Sources Compare?To understand the robustness of our approach, we assess other language models as the knowledge source (i.e. loose teacher): GPT-J [23] and T5-11B adapted for language modelling [10]. We substitute both for GPT-3 as in SS3.2,3.3, generating a small-scale corpus to evaluate. We conduct human evaluation on 1000 examples as above (Table4). Both models attain roughly 72% accuracy, 6 points below GPT-3 (78.5). This suggests strong potential, but higher quality from GPT-3. We explore this further in Appendix B.

 

Making the Teacher More Critical

Symbolic knowledge distillation requires a strong teacher model to maximize the quality of the generated knowledge graph and resulting student model (SS5). While the _loose teacher_ (GPT-3 alone) results in a viable commonsense knowledge graph, evaluation shows this isn't a perfect commonsense teacher. Thus, we multiply in a _critic model_, to filter lower-quality knowledge, _correcting the teacher_ (SS2.1). With modest supervision (a small-scale human evaluation) we train a classifier to predict and discriminate unacceptable examples. We multiply this with the _loose teacher_ SS3, creating a _critical teacher_ product of experts. In practice this means filtering Atomic**10x** to create new corpora that are higher quality, yet still larger scale than human-authored Atomic\({}^{20}_{20}\).

Training a knowledge criticWe gather a training set of _correct vs. incorrect_ human judgments on a randomly-sampled set of 10K entries of Atomic**10x**, as in SS3.4 but with one annotation per example. We take a (random) train/dev/test split of 8k/1k/1k. While this step requires human annotation, humans take on the role of high-level supervisors here-critiquing a small number of generations rather than authoring the entire knowledge graph as in previous work. Indeed, the cost/complexity of this step is similar to a typical human evaluation, making it far cheaper/easier than eliciting human-authored knowledge in past work.

We train binary classifiers (critics) for human acceptability using RoBERTa-Large Liu et al. (2019). We find pretraining on MNLI results in the best model in terms of precision and recall, and we suggest this technique for future studies. We give more detail in Appendix C, including baselines. Our best model vastly improves the accuracy of Atomic**10x** (Table 4), demonstrating that a small amount of human supervision can consistently help to correct GPT-3's mistakes.

Size-accuracy trade-offUsing our critic to filter knowledge results in a natural trade-off between size and accuracy. We test several cutoffs for Atomic**10x**, i.e. confidence at which the critic rejects examples. We report human-measured accuracy (Accept/Reject column Table 4) following SS3.4. We compare the loose teacher (unfiltered) to critical teachers. Discarding 20% of instances that the critic judges as least acceptable (reducing corpus size from 6.5M to 5.1M), Atomic**10x**'s accuracy rises 78.5 \(\rightarrow\) 88.4; human-authored Atomic\({}^{20}_{20}\) contains 600K entries at 86.8% accuracy. Reducing to total size to 2.5M examples (38% of full size), we attain 96.4% accuracy, nearly 10 points above Atomic\({}^{20}_{20}\) while still 4X larger.

What gets filtered out?We qualitatively identify two types of filtered triples: 1) _logical misalignments_, events/inferences joined in an inconsistent manner. Recognizing these requires understanding events-inference interactions, e.g., _X cannot find his shirt **as a result X** is wearing a shirt_; 2) _awkward phrasings_, in which events/inferences are individually incoherent e.g. _PersonX has a fire in the bath_-resulting triples are invalid as the event is implausible.

To understand what is filtered, we ablate the critic (Table 5): our full model is compared to a random predictor, event-only model, and inference-only model. We also compare to an EMAP Hessel and Lee (2020) version, i.e. an ensemble of event and inference-only, without interactions between event/inference (needed for _logical misalignments_).

We find GPT-3 produces both independent awkwardly-phrased events/inferences (filtered by X-only models) and logical misalignments. The classifier, trained on validated knowledge triples, helps in both cases. The EMAP of our full model (identifies only awkward phrases) achieves 87% AP, and our full model (which additionally identifies logical misalignments) improves to 94% AP.

Does filtering hurt diversity?One concern is that the critic may keep only similar "safe" examples, lacking novelty. We repeat our diversity analysis (SS3.4) for critical corpora (Table 4, "Size (div)", higher=better). As we filter, we surprisingly observe proportionally _more_ diverse examples: full Atomic**10x** has a diverse subset 68% of its size; rising to 80% with the most extreme filtering. One possibility is that GPT-3 gravitates towards com

\begin{table}
\begin{tabular}{c|c c c c c} \hline \hline  & Random & Inf & Event & EMAP & Full \\ \hline AP & 79.3 & 81.9 & 86.2 & 87.1 & **94.0** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Average Precision for ablated critic models. The critic not only filters _awkward phrasings_ which can be identified by either the event (**Event**) or inference (**Inf**) in isolation (EMAP only identifies these), but also _logical misalignments_, which require modeling interactions between event/inference, i.e. the full critic (**Full**).

 

mon sentence structures for inconsistent knowledge. These would be recognizable to the critic, and removing them would increase both quality and diversity. This surprising result warrants further study.

## 5 Corpus-to-Machine: Distillation

The final step of symbolic knowledge distillation trains a compact model on the generated natural language knowledge graph. Our base model is GPT2-XL trained on all of Atomic10x: we denote this model by CometDIS. We additionally train the model on critical versions of Atomic10x-critlow denotes training on the corpus achieving 91.5% accuracy, and crithigh on the 96.4% accuracy corpus. Models are trained for 1 epoch, with default parameters using the Huggingface Transformers library Wolf et al. (2019).

Footnote 10: We find Fleiss’ kappa Fleiss (1971) of 47.1 for acceptance, indicating moderate agreement Landis and Koch (1977), and accuracy agreement of 88.7%.

### Evaluating a Symbolically Distilled Model

Evaluation follows past work Hwang et al. (2021); Bosselut et al. (2019); Sap et al. (2019) testing the ability of models to do knowledge base completion, i.e. generating inferences for test events, specifically from the Atomic\({}^{20}_{20}\) test set. We use human evaluation10 following Section 3.4, on 1000 inputs (event + relation), with results in Table 6. We compare to the GPT2-XL-based Comet\({}^{20}_{20}\) model trained on human-generated Atomic\({}^{20}_{20}\), and GPT-3 using the same generation method as SS3-in effect, comparing the student CometDIS to the _loose teacher_ GPT-3. We omit the _critical teacher_ (GPT-3 + critic), which is not assured to produce an inference for each input, as the critic may reject all tails for some inputs. We also compare to zero-shot GPT2-XL Radford et al. (2019) using the same methodology (Table 6).

How does CometDIS compare to GPT-3?In knowledge distillation, the student model often deteriorates in performance Hinton et al. (2015); Kim and Rush (2016) compared to its teacher. Comparing our base teacher-GPT-3-to the simplest version of CometDIS (top-row CometDIS of Table 6) surprisingly shows the student _surpasses_ GPT-3, the model that generates its training data11. We posit that the superior performance of CometDIS may have to do with mistakes of GPT-3 being filtered by verbalization and training of GPT-2, and possibly the focus of ComETDIS on one commonsense domain while GPT-3 covers a more general domain. We leave further study of this effect for future work.

Footnote 11: The slight difference in acceptability for GPT-3 from Table 4 is likely due to variance in raters between rounds of evaluation, and a different distribution of events–Table 4 uses generated events while Table 6 uses events from Atomic\({}^{20}_{20}\).

How does CometDIS compare to human knowledge?While CometDIS without the critic is slightly outperformed by Comet\({}^{20}_{20}\) in terms of accuracy, this reverses with the critic. For both cutoffs tested, CometDIS surpasses Comet\({}^{20}_{20}\), with more filtering resulting in a wider gap.

Usefulness of CometDISFor on-demand inference, where a single high quality inference for some input event/relation is required, CometDIS is the **best available model**: the most performant version surpasses Comet\({}^{20}_{20}\) by 5 points and GPT-3 by over 10. The critical teacher (GPT-3 + critic) yields a more accurate _corpus_, but may filter all inferences for an input, giving no output.

Limits and Future WorkThe success of symbolic knowledge distillation is a first step-demonstrating superior performance to human authoring on the commonsense relations tested here. No aspect of our approach is specific to these relations, yet further work is needed to explore the feasibility of generation for other aspects of commonsense and knowledge, beyond these relations, to concepts like physical or temporal commonsense.

## 6 Related Work

Commonsense Knowledge Graphs (CKG) CKGs provide knowledge for commonsense reasoning. Some are manually constructed, e.g.

\begin{table}
\begin{tabular}{l|c|c c c} \hline \hline CKG Completion & Train Corpus & & & \\ Model & Acc & Accept & Reject & N/A \\ \hline GPT2-XL zero-shot & - & 45.1 & 50.3 & 4.6 \\ GPT-3 & - & 73.3 & 24.1 & 2.6 \\ Comet\({}^{20}_{20}\) & 86.8 & 81.5 & 16.3 & 2.2 \\ \hline CometDIS & 78.5 & 78.4 & 19.2 & 2.4 \\ +criticyww & 91.5 & 82.9 & 14.9 & 2.2 \\ +critichigh & 96.4 & **87.5** & 10.2 & 2.3 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Model performance on knowledge base completion, measured by human judgement. Inferences are generated on held-out events from Atomic\({}^{20}_{20}\). Models besides GPT-3 use GPT-2 XL architecture. CometDIS with a strong critic (+critichigh) achieves the highest acceptance rate overall–87.5.

 

Atomic(Sap et al., 2019; Hwang et al., 2021). ConceptNet (Seper et al., 2017) contains taxonomy and physical commonsense, authored by humans or compiled from such sources. Some CKGs are automatically constructed: TransOMCS (Zhang et al., 2020) extracts 18.48M tuples from syntactic parses and CausalBank Li et al. (2020) extracts 314M cause-effect pairs by pattern-matching. In contrast, we _generate_ commonsense.

Extracting Knowledge from LMsPast work uses models for automatic knowledge graph completion Bosseltut et al. (2019); Hwang et al. (2021); Li et al. (2020). Yet, models are trained on _existing_ resources; Atomic**10x** is generated without these. Other works mine factual/commonsense knowledge directly from off-the-shelf LMs Petroni et al. (2019); Davison et al. (2019); Xiong et al. (2020), but not resulting in the quality at scale of Atomic**10x**.

Knowledge DistillationOther works use knowledge distillation Hinton et al. (2015) for generation. Sanh et al. (2019) follow a label smoothing formulation, while Kim and Rush (2016) follow a similar formulation to us (SS2.1), use the mode of the teacher distribution rather than sampling. Our work is unique in distilling _specific_ information (commonsense) from a general language model.

Data GenerationWhile manual dataset creation is expensive and complex Schwartz et al. (2017); Agrawal et al. (2018); Tsuchiya (2018); Bras et al. (2020),crowdsourcing is the most popular method for goal-oriented, high quality/coverage datasets.

Past automatic data mainly use extractive approaches, e.g. syntactic parsing Zhang et al. (2020) or pattern matching Li et al. (2020) from unstructured text Lehmann et al. (2015); Buck et al. (2014). These scale, but are noisy and limited in format-atomic knowledge will not appear simply in natural text. Some works explore automatic data synthesis/expansion by finetuning LMs on existing labeled data Anaby-Tavor et al. (2020); Papanikolaou and Pierleoni (2020); Kumar et al. (2020); Yang et al. (2020), but are limited by data quality.

## 7 Conclusions

We introduce symbolic knowledge distillation, a _machine-to-corpus-to-machine_ pipeline for commonsense that does not require human-authored knowledge--instead, using machine generation. Knowledge is transferred from a large, general model to a compact commonsense model, through a commonsense corpus-yielding a commonsense knowledge graph and model. Our resulting symbolic knowledge graph has greater scale, diversity, and quality than human authoring. symbolic knowledge distillation offers an alternative to human-authored knowledge in commonsense research.

## Acknowledgments

This work was funded in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) (funding reference number 40123309), DARPA MCS program through NIWC Pacific (N66001-19-2-4031), and the Allen Institute for AI.

## Ethical Considerations

One aspect of our work with the potential for ethical pitfalls is large-scale generation from pretrained language models, in constructing Atomic**10x**. Recent work Bender et al. (2021) has highlighted the risks of models trained on massive text resources, as GPT-3 Brown et al. (2020) is, which we use for generation. Indeed, open generations from pre-trained language models can often contain harmful, biased, or offensive aspects. We argue here that this risk is largely mitigated in our work, mainly due to the narrow and constrained nature of our generations. The goal of our work is characterising simple and generic anonymous situations, specifically in terms of commonsense causes and effects. We ensure generations are focused on these topics through careful prompting, which we found to be quite effective at keeping these generations ontopic. As such, the potential for harmful generation is very low; indeed, in a manual inspection of 100 generated examples, we found none that were significant harmful, besides one that contained adult content.

A related concern is the potential for large models and training sets to make automated compression or exploitation possible, for instance in surveillance or generating fake news. As above, we argue that the generic, commonsense nature of our data and models makes this concern less relevant here. Our data does not contain any information directly related to these harmful domains (e.g. social media or fake news generation). While our data may assist machines in understanding basic situations, this is unlikely to be useful for harmful models given the simplicity of our data and still-flawed commonsense capabilities of even the most advanced 

models.

Finally, we note that we ensure fair and generous compensation for all human evaluators we hire through Amazon Mechanical Turk. Based on our estimates of time required per task, we ensure that the effective pay rate is at least $15 per hour.

## References

* A. Agrawal, D. Batra, P. Parikh, and A. Kembhavi (2018)Don't just assume; look and answer: overcoming priors for visual question answering. Note: _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 4971-4980 Cited by: SS3.
* P. Ammanabrolu, W. Cheung, W. Broniec, and M. O. Riedl (2021a)Automated storytelling via causal, commonsense plot ordering. In AAAI, Cited by: SS1, SS2.
* P. Ammanabrolu, J. Urbanek, M. Li, C. D. Szlam, T. Rocktaschel, and J. Weston (2021b)How to motivate your dragon: teaching goal-driven agents to speak and act in fantasy worlds. In NAACL, Cited by: SS1, SS2.
* A. Anaby-Tavor, B. Carmeli, E. Goldbraich, A. Kantor, G. Kour, S. Shlomov, N. Tepper, and N. Zwerdling (2020)Do not have enough data? deep learning to the rescue!. Proceedings of the AAAI Conference on Artificial Intelligence34, pp. 7383-7390. External Links: Link, Document Cited by: SS1, SS2.
* F. Arabshahi, J. Lee, A. Bosselut, Y. Choi, and T. Mitchell (2021)Conversational multi-hop reasoning with neural commonsense knowledge and symbolic logic rules. In EMNLP, Cited by: SS1, SS2.
* E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell (2021)On the dangers of stochastic parrots: can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, New York, NY, USA, pp. 610-623. External Links: ISBN 978-1-4503-3213-1, Link, Document Cited by: SS1, SS2.
* S. Bhakthavatsalam, C. Anastasiades, and P. E. Clark (2020)Genericskb: a knowledge base of generic statements. ArXivabs/2005.00660. Cited by: SS1, SS2.
* K. D. Bollacker, C. Evans, P. K. Paritosh, T. Sturge, and J. Taylor (2008)Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD Conference, Cited by: SS1, SS2.
* A. Bosseltu, H. Rashkin, M. Sap, C. Malaviya, A. Celikyilmaz, and Y. Choi (2019)Comet: commonsense transformers for automatic knowledge graph construction. In ACL, Cited by: SS1, SS2.
* R. L. Bras, S. Swayamdipta, C. Bhagavatula, Rowan Zellers, Matthew E. Peters, A. Sabharwal, and Y. Choi (2020)Adversarial filters of dataset biases. In ICML, Cited by: SS1, SS2.
* T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language models are few-shot learners. External Links: 2004.02201 Cited by: SS1, SS2.
* C. Buck, K. Heafield, and B. Van Ooyen (2014)N-gram counts and language models from the common crawl. In LREC, Vol. 2, pp. 4. Cited by: SS1, SS2.
* T. Chakrabarty, D. Ghosh, S. Muressan, and N. Peng (2020)R^3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge. In ACL, Cited by: SS1, SS2.
* T. Chakrabarty, X. Zhang, S. Muresan, and N. Peng (2021)MERMAID: metaphor generation with symbolism and discriminative decoding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 4250-4261. External Links: Link, Document Cited by: SS1, SS2.
* Ernest Davis and Gary Marcus (2017)Causal generative models are just a start. Behavioral and Brain Sciences40. Cited by: SS1, SS2.
* J. Davison, J. Feldman, and A. M. Rush (2019)Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1173-1178. Cited by: SS1, SS2.
* O. Etzioni, A. Fader, J. Christensen, S. Soderland, et al. (2011)Open information extraction: the second generation. In Twenty-Second International Joint Conference on Artificial Intelligence, Cited by: SS1, SS2.
* J. L. Fleiss (1971)Measuring nominal scale agreement among many raters. Psychological bulletin76 (5), pp. 378. Cited by: SS1, SS2.
* J. Hessel and L. Lee (2020)Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think!. In EMNLP, Cited by: SS1, SS2.
* Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean (2015)Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, Cited by: SS1, SS2.
* Geoffrey E Hinton (2002)Training products of experts by minimizing contrastive divergence. Neural computation14 (8), pp. 1771-1800.

 

* Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Hwang et al. (2021) Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. _AAAI_.
* Kearns et al. (2020) William R. Kearns, Neha Kaura, Myra Divina, Cuong Viet Vo, Dong Si, Teresa M. Ward, and Weichao Yuwen. 2020. A wizard-of-oz interface and persona-based methodology for collecting health counseling dialog. _Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems_.
* Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In _EMNLP_.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. _CoRR_, abs/1412.6980.
* Kumar et al. (2020) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained transformer models. In _Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems_, pages 18-26, Suzhou, China. Association for Computational Linguistics.
* Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. _biometrics_, pages 159-174.
* a large-scale, multilingual knowledge base extracted from wikipedia. _Semantic Web_, 6:167-195.
* Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In _EMNLP_.
* Li et al. (2020) Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, and Benjamin Van Durme. 2020. Guided generation of cause and effect. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20._
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Merrill et al. (2021) William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. 2021. Provable limitations of acquiring meaning from ungrounded form: What will future language models understand? _Transactions of the Association for Computational Linguistics_, 9:1047-1060.

* Papankolaou and Pierleoni (2020) Yannis Papankolaou and A. Pierleoni. 2020. Dare: Data augmented relation extraction with gpt-2. _ArXiv_, abs/2004.13845.
* Petroni et al. (2019) Fabio Petroni, Tim Rocktaschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2463-2473.
* Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
* Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. _ArXiv_, abs/1910.01108.
* Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019. Atomic: An atlas of machine commonsense for ifthen reasoning. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 3027-3035.
* Schwartz et al. (2017) Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the ROC story clozse task. In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_, pages 15-25, Vancouver, Canada. Association for Computational Linguistics.
* Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 31.
* Talmor et al. (2021) Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. 2021. Commonsenseqa 2.0: Exposing the limits of ai through gamification.
* Tsuchiya (2018) Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In _Proceedings of the Eleventh International Conference on Language Resources and 

Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
* Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax).
* Welleck et al. (2020) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In _International Conference on Learning Representations_.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122. Association for Computational Linguistics.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. _ArXiv_, abs/1910.03771.
* Xiong et al. (2020) Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In _International Conference on Learning Representations_.
* Yang et al. (2020) Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. Generative data augmentation for commonsense reasoning. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 1008-1025, Online. Association for Computational Linguistics.
* Zhang et al. (2020a) Hongming Zhang, Daniel Khashabi, Yong, and D. Roth. 2020a. TransOMCS: From linguistic graphs to commonsense knowledge. In _IJCAI_.
* Zhang et al. (2020b) Hongming Zhang, Xin Liu, Haojie Pan, Y. Song, and C. Leung. 2020b. Aser: A large-scale eventuality knowledge graph. _Proceedings of The Web Conference 2020_.
* Zhou et al. (2020) Ben Zhou, Qiang Ning, Daniel Khashabi, and Dan Roth. 2020. Temporal common sense acquisition with minimal supervision. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7579-7589, Online. Association for Computational Linguistics.
* Zhu et al. (2018) Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 1097-1100 

Human Evaluation Details

We conduct human evaluations on Amazon Mechanical Turk using the template of Figures 4,5. Workers are presented with Atomic-style triples, replacing relations with natural language templates (e.g. _HinderedBy_ becomes "can be hindered by"). 3 annotators rate each triple, with options for acceptability: "always/often", "sometimes/likely", "farfetched/never", "invalid", or "too unfamiliar to judge". The first two are considered "accepted", the second two "rejected" and the final is "no judgement". For reporting acceptance rates, and training a critic model, we only distinguish between "accepted" and not "accepted".

Workers are compensated $0.17 per task (i.e. completing all questions in the evaluation template Figures 4,5). We estimate an upper bound of 30s to complete a single task, which gives an hourly rate of $20.4. Workers are selected based on an Amazon Mechanical Turk qualification, specifically filtering for workers with high accuracy on past knowledge base triple evaluations. We follow the same setup for all evaluations, besides number of annotators. This setup is shown to result in consistent and reliable annotations, with an inter-annotator agreement given by Fleiss' kappa [6] of 40.8 when evaluating with 3 annotators, in SS3.4.

## Appendix B Using Alternate Models as Knowledge Sources

One natural question that arises from the strong performance of symbolic knowledge distillation is whether other sources of knowledge (i.e. language models) would similarly benefit from this method. In this section, we particularly measure the capacity of other language models to serve as the "loose teacher" which generated the base knowledge of the resulting corpus.

We expand our study beyond GPT-3 here (the model used in our work), to include 2 contemporary large language models, GPT-J [9] and T5-11B [11] finetuned for language modelling. For knowledge generation (verbalization) we follow the same procedure as SS3 along with simple adjustments to improve quality. We are investigating the effect of the critic on knowledge precision here, so we also include Atomic\({}^{20}_{20}\) to probe the usefulness of automatic filtering for human-authored knowledge.

For each knowledge source, we follow the human evaluation setup in SS3.4 to obtain quality annotations of 2000 examples, with 1 annotation per example. This follows a similar setup to SS4--indeed, we are replicating the earlier critic experiments but at a smaller scale (2000 annotations vs. 10000) to allow for more knowledge sources. For each knowledge source, we randomly split into sizes of 1400/300/300 for train, dev, and test sets. We follow SS4 to train a critic model for each knowledge source.

We plot different thresholds (% of corpus filtered) against the resulting precision (proportion of corpus that is judged to be "valid" knowledge) in Figure 3, and give numbers at various sizes in Table 7. One striking aspect is that a critic model can raise the precision of any of these knowledge sources to approximately 90% while retaining 30% of the original corpus size. While this discards a significant portion of the original generated knowledge, it raises the exciting prospect of using more cost-effective models at a large scale to generate strong commonsense corpora like Atomic**10x**. GPT-J and T5-11B can both be run locally by researchers, unlike GPT-3 which uses a pay-per-generation API. Thus, one can imagine producing a large and high-quality corpus like Atomic**10x** at a lower cost by instead generating a larger volume of knowledge from such an accessible model, and simply filtering to a greater extent.

Another interesting aspect is how the various knowledge sources diverge. Under little to no critical filtering (i.e. corpus size = 1.0), the precision of various knowledge sources is widely spread. Before applying a critic, quality of knowledge source is very important. Indeed, precision is ordered by cost of generation: human Atomic\({}^{20}_{20}\) has the highest precision while being the most expensive, followed by GPT-3 (used here) which is pay-per-generation, and finally the two publicly available models. Another point of divergence is for extreme filtering (at approximately 20% of the original corpus size. All knowledge sources but GPT-3 plateau at approximately 90% accuracy, while GPT-3 rises towards 100%. Indeed, this supports our use of GPT-3 in this work, as a high-quality automatic knowledge source.

## Appendix Critic Model

We train binary classifiers (critics) for human acceptability using RoBERTa-Large [10], fine-tuning all parameters, along with a 2-layer MLP on the [CLF] representation. We conduct 

a small grid search on the validation set finding batch size 128, dropout .1, and Adam (Kingma and Ba, 2015) learning rate 5e-6 to be effective. We use early stopping and decay learning rate on validation performance plateauing, to maximize \(R@80\%\) on the validation set. We find RoBERTa pretrained on MNLI Williams et al. (2018) effective, outperforming other options. As well, we substitute randomly-sampled names in for person designations "X"/"Y". We include as a baseline an unsupervised filtration metric inspired by Davison et al. (2019): they propose a model estimate of PMI to score mined commonsense triples. In our case, we use Negative Log-Likelihood (NLL) and token-mean-NLL from GPT-3 itself.

The validation precision/recall of our best performing model, the baselines, and the in-optimal hyperparameter configurations are given in Figure 6. Once fixing our model, we applied it to the test set (also in Fig 6), verifying that it generalizes to Atomic\({}^{\mathbf{lbx}}\) entries. Overall, our trained critic model is more effective than the baselines in identifying high and low quality teacher generations at all levels of precision and recall. This result demonstrates that a small amount of human supervision can consistently help to correct GPT-3's mistakes.

## Appendix D Atomic\({}^{\mathbf{lbx}}\) Generation Prompts

We include example prompts for all generations we do, from Table 8 to 15. Note that elements of generation prompts are randomized for each batch. For event generation, the few-shot examples and order are randomly sampled from a seed set of 100 high-quality examples from Atomic\({}_{20}^{20}\) in each batch. For inference generation, the natural names used for PersonX and PersonY are randomly sampled from a small predefined set of names.

Figure 3: Precision resulting from the critic step from §4, with various thresholds. We include corpora generated by GPT-3 (Atomic\({}^{\mathbf{lbx}}\)), GPT-T5-11B, and humans (Atomic\({}_{20}^{20}\)). Without filtering (corpus size = 1.0), different corpora have a variety of precisions. As more examples are filtered by the critic, precision rises significantly demonstrating the strong value of the critic step.

\begin{table}
\begin{tabular}{l|c c c c c c c c c c} \hline \hline  & \multicolumn{7}{c}{**Precision** at Corpus Size (\%)} \\ Knowledge Source & 100 & 90 & 80 & 70 & 60 & 50 & 40 & 30 & 20 & 10 \\ \hline Atomic\({}_{20}^{20}\) & 84.0 & 86.3 & 87.9 & 89.0 & 88.3 & 88.7 & 91.7 & 90.0 & 90.0 & 90.0 \\ GPT-J & 71.7 & 76.7 & 81.7 & 83.8 & 86.7 & 88.0 & 88.3 & 87.8 & 93.3 & 90.0 \\ T5-11B & 64.7 & 66.7 & 70.8 & 74.8 & 79.4 & 84.7 & 89.2 & 92.2 & 91.7 & 93.3 \\ GPT-3 curie & 79.3 & 81.5 & 85.0 & 86.2 & 88.3 & 90.7 & 91.7 & 90.0 & 98.3 & 100.0 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Knowledge precision at various corpus sizes (from 100% to 10%) based on filtering by the critic model. Precision is calculated by human annotation of valid or invalid knowledge. We consider 4 knowledge sources, as described in Appendix B. This corresponds to the data plotted in Figure 3.

 

* [150] M. C. B. K., and R. J. S. McAllister. A survey of the data: a survey of the data. In _Proceedings of the 27th ACM SIGKDD international conference on Knowledge discovery and data mining_, pages 34-68, 2009.
* [151] D. L. Miller. _The Data Mining_. Addison-Wesley Professional Company, New York, NY, USA, 1997.
* [152] G. P. T. Jones. _A Survey of Data Mining_. Addison-Wesley Professional Company, New York, NY, USA, 1997.
* [153] H. W. Anderson. _Data Mining_. Addison-Wesley Professional Company, New York, NY, USA, 1997.

[MISSING_PAGE_POST]

 

* [25] M. C. Bader, A. K. Goyal, and R. J. Olver. The role of the social network in a social context: an empirical study. _Social Networks_, 19(3):467-480, 2000.
* [26] D. E. Kingma and S. Welling. Visualizing data using deep neural networks. In _Advances in Neural Information Processing Systems_, pages 1398-1403, 2014.
* [27] P. L. LeCun, Y. Bengio, and T. Darrell. Learning for large scale representations by jointly learning. In _Proceedings of the Thirtieth International Conference on Machine Learning_, pages 139-146, 2014.
* [28] H. Li, X. Zhang, Z. Liu, and W. Dong. Deep residual learning with convolutional neural networks. In _NIPS_, volume 1, pages 1739-1746, 2014.

[MISSING_PAGE_POST]

 

* [1] A. Krizhevsky, S. Bengio, and J. Schmidhuber. Learning deep neural networks with applications in natural language processing. In _NIPS_, pages 239-240, 2015.
* [2] M. L. Bottou, Y. Bengio, and D. Salakhutdinov. Generative adversarial nets for large data sets. In _Advances in Neural Information Processing Systems_, volume 67 of _Lecture Notes in Computer Science_, pages 83-90, 2015.
* [3] R. B. C. Barnett, P. G. Beals, and T. W. Teh. Deep learning on the image database. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 103-113, 2015.
* [4] H. Chen, X. Li, and Z. Zhang. Deep residual learning for image recognition. In _CVPR_, pages 1098-1099, 2015.
* [5] I. Goodfellow, I. Sutskever, and J. Shlens. Imagenet classification by jointly connected layers. In _NeurIPS_, pages 1098-1110, 2015.
* [6] V. Khoslavi, N. Mirza, and E. Rabinovich. Very accurate representations of tiny images. In _ICLR_, pages 1098-1110, 2015.

[MISSING_PAGE_POST]

 

* [18] S. K. Bender, A. M. C. R. Bishop, and J. L. Friedman. The role of the word's words in a database. In _Proceedings of the 20th ACM SIGGRAPH Conference on Knowledge Discovery and Data Mining_, pages 35-46, New York, NY, USA, 2007.

[MISSING_PAGE_POST]

 

Next, what do situations make people do? Examples:

\begin{tabular}{l l} \hline  & Situation 1: Devin gets a divorce. \\ As a result, Devin dates someone new. \\ Situation 2: Jamie lifts weights. \\ As a result, Jamie has sore muscles. \\ Situation 3: Sydney takes Ryan to a bar. \\ As a result, Sydney gets drunk. \\ Situation 4: Lindsay decides to hire a tutor. \\ As a result, Lindsay gets better grades. \\ Situation 5: Rowan buys Pat drinks. \\ As a result, Rowan is thanked by Pat. \\ Situation 6: Lee hears bad news. \\ As a result, Lee begins to cry. \\ Situation 7: Riley buys a chocolate bar. \\ As a result, Riley gets change. \\ Situation 8: Adrian does a lot of work. \\ As a result, Adrian gets mental fatigue. \\ Situation 9: Hunter attends a concert. \\ As a result, Hunter hears a new song. \\ Situation 10: Sam gets the job done. \\ As a result, Sam gets more responsibilities. \\ Situation 11: Alex makes Chris wait. \\ As a result, Alex \\ \hline \end{tabular} Table 10: Prompt for generating xEffect.

 

For each situation, describe the intent. Examples:

* [leftmargin=0cm]
* Situation 1: Devin gets the newspaper.
* Devin intends to read the newspaper.
* Situation 2: Jamie works all night.
* Jamie intends to meet a deadline.
* Situation 3: Sydney destroys Ryan.
* Sydney intends to punish Ryan.
* Situation 4: Lindsay clears her mind.
* Lindsay intends to be ready for a new task.
* Situation 5: Rowan wants to start a business.
* Rowan intends to be self sufficient.
* Situation 6: Lee ensures Ali's safety.
* Lee intends to be helpful.
* Situation 7: Riley buys lottery tickets.
* Riley intends to become rich.
* Situation 8: Alex makes Chris wait.
* Alex intends

\begin{table}
\begin{tabular}{l l} \hline \hline For & each situation, describe the intent. Examples: \\ \hline  & Situation 1: Devin gets the newspaper. \\ \multicolumn{2}{l}{Devin intends to read the newspaper.} \\ \multicolumn{2}{l}{Situation 2: Jamie works all night.} \\ \multicolumn{2}{l}{Jamie intends to meet a deadline.} \\ \multicolumn{2}{l}{Situation 3: Sydney destroys Ryan.} \\ \multicolumn{2}{l}{Sydney intends to punish Ryan.} \\ \multicolumn{2}{l}{Situation 4: Lindsay clears her mind.} \\ \multicolumn{2}{l}{Lindsay intends to be ready for a new task.} \\ \multicolumn{2}{l}{Situation 5: Rowan wants to start a business.} \\ \multicolumn{2}{l}{Rowan intends to be self sufficient.} \\ \multicolumn{2}{l}{Situation 6: Lee ensures Ali’s safety.} \\ \multicolumn{2}{l}{Lee intends to be helpful.} \\ \multicolumn{2}{l}{Situation 7: Riley buys lottery tickets.} \\ \multicolumn{2}{l}{Riley intends to become rich.} \\ \multicolumn{2}{l}{Situation 8: Alex makes Chris wait.} \\ \multicolumn{2}{l}{Alex intends} \\ \hline \hline \end{tabular}
\end{table}
Table 11: Prompt for generating xIntent.

 

Next, we will discuss what people need for certain situations. Examples:

1. Before Devin makes many new friends, Devin has to spend time with people.
2. Before Jamie gets a date, Jamie has to ask someone out.
3. Before Sydney changes Ryan's mind, Sydney has to think of an argument.
4. Before Lindsay gets a job offer, Lindsay has to apply.
5. Before Rowan takes a quick nap, Rowan has to lie down.
6. Before Lee tries to kiss Ali, Lee has to approach Ali.
7. Before Riley rides Noel's skateboard, Riley has to borrow it.
8. Before Adrian eats the food, Adrian has to prepare a meal.
9. Before Hunter watches Netflix, Hunter has to turn on the TV.
10. Before Sam has a baby shower, Sam has to invite some friends.
11. Before Alex makes Chris wait, Alex has

\begin{table} \end{table}
Table 12: Prompt for generating xNeed.

 

Next, how do people feel in each situation? Examples:

* [1]
* [2] Situation 1: Devin lives with Jean's family.
* [3] Devin feels loved.
* [4] Situation 2: Jamie expects to win.
* [5] Jamie feels excited.
* [6] Situation 3: Sydney comes home late.
* [7] Sydney feels tired.
* [8] Situation 4: Lindsay sees dolphins.
* [9]Lindsay feels joyful.
* [10] Situation 5: Rowan causes Pat anxiety.
* [11] Rowan feels guilty.
* [12] Situation 6: Lee goes broke.
* [13] Lee feels embarrassed.
* [14] Situation 7: Riley has a drink.
* [15] Riley feels refreshed.
* [16] Situation 8: Adrian has a heart condition.
* [17] Adrian feels scared about their health.
* [18] Situation 9: Hunter shaves Avery's hair.
* [19] Hunter feels helpful.
* [20] Situation 10: Sam loses all of Charlie's money.
* [21] Sam feels horrible.
* [22] Situation 11: Alex makes Chris wait.
* [23] Alex feels

\begin{table} \end{table}
Table 13: Prompt for generating xReact.

 

Next, what do people want in each situation? Examples:

\begin{tabular}{l l} \hline \hline  & Situation 1: Devin mows the lawn. \\ \multicolumn{2}{l}{Devin wants to take a shower.} \\ \multicolumn{2}{l}{Situation 2: Jamie is going to a party.} \\ \multicolumn{2}{l}{Jamie wants to take an Uber home.} \\ \multicolumn{2}{l}{Situation 3: Sydney bleeds a lot.} \\ \multicolumn{2}{l}{Sydney wants to go to the ER.} \\ \multicolumn{2}{l}{Situation 4: Lindsay works as a cashier.} \\ \multicolumn{2}{l}{Lindsay wants to find a better job.} \\ \multicolumn{2}{l}{Situation 5: Rowan gets dirty.} \\ \multicolumn{2}{l}{Rowan wants to do a load of laundry.} \\ \multicolumn{2}{l}{Situation 6: Lee stays up all night studying.} \\ \multicolumn{2}{l}{Lee wants to rest.} \\ \multicolumn{2}{l}{Situation 7: Riley gets Noel's autograph.} \\ \multicolumn{2}{l}{Riley wants to tell some friends.} \\ \multicolumn{2}{l}{Situation 8: Adrian sees Taylor's point.} \\ \multicolumn{2}{l}{Adrian wants to agree with Taylor.} \\ \multicolumn{2}{l}{Situation 9: Hunter leaves Avery's bike.} \\ \multicolumn{2}{l}{Hunter wants to keep the bike safe.} \\ \multicolumn{2}{l}{Situation 10: Sam wants a tattoo.} \\ \multicolumn{2}{l}{Sam wants to find a tattoo design.} \\ \multicolumn{2}{l}{Situation 11: Alex makes Chris wait.} \\ \multicolumn{2}{l}{Alex wants} \\ \hline \hline \end{tabular} Table 14: Prompt for generating xWant.

 

Next, what can hinder each situation? Examples:

Situation 1: Devin makes a doctor's appointment,

This is hindered if Devin can't find the phone to call the doctor.

Situation 2: Jamie rubs Wyatt's forehead,

This is hindered if Jamie is afraid to touch Wyatt.

Situation 3: Sydney eats peanut butter,

This is hindered if Sydney is allergic to peanuts.

Situation 4: Lindsay looks perfect,

This is hindered if Lindsay can't find any makeup.

Situation 5: Rowan goes on a run,

This is hindered if Rowan injures her knees.

Situation 6: Lee takes Ali to the emergency room,

This is hindered if Ali has no health insurance to pay for medical care.

Situation 7: Riley spends time with Noel's family,

This is hindered if Noel's family doesn't like spending time with Riley.

Situation 8: Adrian moves from place to place,

This is hindered if Adrian can't afford to move.

Situation 9: Hunter protests the government,

This is hindered if Hunter is arrested.

Situation 10: Sam has a huge fight,

This is hindered if Sam does not like confrontation.

Situation 11: Alex makes Chris wait,

This is hindered if

\begin{table}
\begin{tabular}{l l} \hline \hline Next, what can hinder each situation? Examples: \\ Situation 1: Devin makes a doctor’s appointment, \\ This is hindered if Devin can’t find the phone to call the doctor. \\ Situation 2: Jamie rubs Wyatt’s forehead, \\ This is hindered if Jamie is afraid to touch Wyatt. \\ Situation 3: Sydney eats peanut butter, \\ This is hindered if Sydney is allergic to peanuts. \\ Situation 4: Lindsay looks perfect, \\ This is hindered if Lindsay can’t find any makeup. \\ Situation 5: Rowan goes on a run, \\ This is hindered if Rowan injures her knees. \\ Situation 6: Lee takes Ali to the emergency room, \\ This is hindered if Ali has no health insurance to pay for medical care. \\ Situation 7: Riley spends time with Noel’s family, \\ This is hindered if Noel’s family doesn’t like spending time with Riley. \\ Situation 8: Adrian moves from place to place, \\ This is hindered if Adrian can’t afford to move. \\ Situation 9: Hunter protests the government, \\ This is hindered if Hunter is arrested. \\ Situation 10: Sam has a huge fight, \\ This is hindered if Sam does not like confrontation. \\ Situation 11: Alex makes Chris wait, \\ This is hindered if \\ \hline \hline \end{tabular}
\end{table}
Table 15: Prompt for generating HinderedBy.