NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation
Authors: Peter West
Ronan Le Bras
Taylor Sorensen
Bill Yuchen Lin
Liwei Jiang
Ximing Lu
Khyathi Chandu
Jack Hessel
Ashutosh Baheti
Chandra Bhagavatula
Yejin Choi
Venue: EMNLP 2023 Findings (2023)
Keywords: Knowledge
Commonsense
Distillation
Model
Symbolic
TL;DR: We use symbolic knowledge distillation to produce NovaCOMET, a high-quality, open-format knowledge model

Abstract: We present NovaCOMET, an open commonsense knowledge model, that combines the best aspects of knowledge and general task models. Compared to previous knowledge models, NovaCOMET allows open-format relations enabling direct application to reasoning tasks; compared to general task models like Flan-T5, it explicitly centers knowledge, enabling superior performance for commonsense reasoning. 

NovaCOMET leverages the knowledge of opaque proprietary models to create an open knowledge pipeline. First, knowledge is symbolically distilled into NovATOMIC, a publicly-releaseddiscrete knowledge graph which can be audited, critiqued, and filtered.  Next, we train NovaCOMET on NovATOMIC by fine-tuning an open-source pretrained model. NovaCOMET uses an open-format training objective, replacing the fixed relation sets of past knowledge models, enabling   arbitrary structures within the data to serve as inputs or outputs. 

The resulting generation model, optionally augmented with human annotation, matches or exceeds comparable open task models like Flan-T5 on a range of commonsense generation tasks. NovaCOMET serves as a counterexample to the contemporary focus on instruction tuning only, demonstrating a distinct advantage to explicitly modeling commonsense knowledge as well.

NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillation
Peter West Ronan Le Bras*
Taylor Sorensen
Bill Yuchen Lin Liwei Jiang Ximing Lu Khyathí Chandu²
Jack Hessel Ashutosh Bahet Chandra Bhagavatula* Yejin Choi 1#
Paul G. Allen School of Computer Science & Engineering, University of Washington Allen Institute for Artificial Intelligence
pawestêcs. washington. edu
Abstract
Closed LLM
A
Open Data
i
Open Model
We present NOVACOMET, open common- sense knowledge model, that combines the best aspects of knowledge models and general task models. Compared to previous knowledge models, NOVACOMET allows open-format re- lations enabling direct application reason- ing tasks; compared to general task models Flan-T5, NOVACOMET explicitly centers knowledge, enabling superior performance for commonsense reasoning.
NOVATOMIC
NovaCOMET
Going to School [HASK]
Open Format
What should you bring? Lunch, textbaoks,
[NASK]
stationary
[MASK]
Eliza is playing in school
Mho is there besides Eliza? Parents, referees
game
We leverage opaque-bue wes---uutt---t-ee powerful proprictary LLMs into open commonsense pipeline by: creat- ing an auditable knowledge base NOVATOMIC that gives fine tne-grained control over included knowledge, (ii) en- suring the generated knowledge uses cigher-coverage open-format with natural-language queries as relations and flexible mask-filling to allow for more open com- monsense use-cases, (iii) demonstrating the effective- ness of (i) and (ii) via NOvACOMET's superior perfor- number of tasks, under both automatic and human evaluations.
Figure I:
NOVACOMET leverages the knowledge of opaque proprietary models to create an open knowledge pipeline. First, knowledge is sym- bolically distilled into NOVATOMIC, a publicly- released' discrete knowledge graph which can be audited, critiqued, and filtered. Next, train NOVACOMET on NOvATOMIC by fine- tuning open-source pretrained model. No- VACOMET uses an open-format training ob- jective, replacing fixed relation sets past knowledge models, enabling arbitrary struc- tures within the data to serve as inputs or out- puts.
models to be open task solvers (e.g. instruction tun- ing) we find that explicitly modeling knowledge in NOVACOMET also provides distinct advantage, with NOVACOMET showing similar superior performance comparable open task models on a range of commonsense reasoning benchmarks.
The resulting generation model, optionally aug- mented with human annotation, matches or ex- ceeds comparable open task models like Flan- T5 on range of commonsense generation tasks. NOVACOMET serves as a counterex- ample to the contemporary focus on instruction tuning only, demonstrating distinct advantage to explicitly modeling commonsense knowl- edge as well.
For NOVACOMET, we leverage opaque, pro- prictary models like ChatGPT or GPT-4 (Ouyang et al., 2022; OpenAI, 2023) as the knowledge source in an open commonsense pipeline (Figure 1), Such models have demonstrated remarkable com- monsense ability (Bubeck et al., 2023; Bian et al.. 2023) yet, closed and opaque, their direct useful- ness for studying commonsense limited. Without information about training or direct access the model, it impossible to study where reported gains come from-c.g. c.g. the extent of test set con- tamination with benchmarks.
1 Introduction
We present NOVACOMET, open commonsense knowledge model combining the advantages of both knowledge models and general task models, NOVACOMET models commonsense knowledge with an open format, allowing to applied to general reasoning tasks contrast to previous knowledge models. Compared to simply training
Our resources are available at novacomet.des
In our work, we use these models first to gener-

---

ale an open knowledge base (NoVATOMIC, $2.1), which can be analyzed, improved, and verified against test set contamination. Next, we train an open commonsense model (NovACOMET, $2.3) on this knowledge: the underlying data and code will be released along with the model for the study of commonsense. This allows future testing of NOvACOMET (and of other models based on No- VATOMIC) analyze the training set set-essentially allowing us to distill information from base LLM into an auditable format.
sense reasoning.
NOvACOMET trained with symbolic knowl- edge distillation (West et al., 2021) by combin- ing the commonsense data generated by large lan- guage models ($2.1) with high-quality annotations of knowledge plausibility ($2.2). We experiment with multiple methods for combining generated data with plausibility information (indicating how likely given knowledge is) train the final model, NOVACOMET ($2.3).
In training NOVACOMET, we also use an open format: compared to previous knowledge models which use fixed relation set and training order (head relation- tail) we natural language queries as relations, and allow masked generation of all aspects of the data. This allows our model to used in a wide range of general reasoning tasks, thus addressing a significant limitation of prior knowledge models that are limited to down- stream upplications capable of effectively leverag- ing their restricted set of relations. Enabling an open format also allows the knowledge generation focus on pertinent aspects of the context, rather than forcing the generation of inferences for arbi- trary, potentially irrelevant relations.
2.1 Generating Open Commonsense Data
Following symbolic knowledge distillation (West et al., 2021), we distill large quantities of high- quality knowledge from very large, general founda- tion models ($2.1.1) - we call the resulting dataset NOVATOMIC. One major difference from previous knowledge graphs is that we allow an open rela- tion set, in the form of queries rather than fixed relation tokens. While commonsense knowledge often takes a head, relation, tail format with a fixed set of discrete relations (c.g. X buys a lot- tery ticket, xWant, to win.), we propose a context, query, inference (CQI) format with natural lan- guage queries serving open relations. We also analyze unique properties of this distilled knowl- edge in $2.1.2.
Following past work on symbolic knowledge distillation (West et al., 2022), we also use No- VATOMIC as the basis for training plausibility model with human annotations ($2.2), and study how this improve NOVACOMET ($2.3).
2.1.1 Data Generation
We outline the generation process below, which consists of (1) generating contexts and (2) generat- ing queries/inferences, resulting in our new knowl- edge base, NOVATOMIC.
We test NOVACOMET on a range of common- sense generation tasks, and find that it consistently outperforms general task models of comparable size, such as Flan-T5zzl (Chung et al., 2022a) and TO commonsense tasks like abductive infilling and explanation generation. Furthermore, we as- sess the ability of our plausibility model handle general commonsense QA tasks and observe that it achieves comparable or superior discriminative performance on a range tasks. NOVACOMET will serve as an open resource for studying com- monsense, and an example of the advantage of explicitly modeling commonsense knowledge in contrast to general task modeling alone.
Context Generation. First, we have experts gen- erate 21 varied prompts to steer models to gen- erate events or situations that require common- sense knowledge to fully understand (see B.1 for all prompts used). As variations in prompt wording influence the model's I's output, prompts enhance both diversity and coverage of the generated outputs. Half of the time, we generate contexts zero-shot manner, while for the other half, we do one-shot generation with one example drawn from ATOMIC 10 (West et al., 2022). order to reduce generation cost, we generate 20 situations per prompt (batched generation).
many different
2 NOVACOMET: open commonsense models
We generate the contexts using GPT-3 (Brown et al., 2020) variant text-davinci-003 (Ouyang et al., 2022) for total cost of USD $39.56. We set top_p=0. 99 and presence__enaltyyaee ing the logit values for tokens that have already
NOVACOMET a large-scale, open common- sense model that can handle both explicit knowl- edge generation, and tasks that require common-
lower-

---

occurred promote diversity within each batch. Finally, to allow NOvACOMET to see some di- versity names, we also randomly swap all enti- ties (names or "PersonX/Y/Z"") for a name drawn from the 2021 public US social security application name registry² with probability 0.5.
Dataset ATOMIC
Entries #
3-grams
Type
Context
2020
43,958
3.5
40,194
55,8
&
10x
165,783
2.6
235,172
44,9
Event
NOVA
102,195
4.7
343,636
44.6
Query &
2020
23
10X
7
Relation
NOVA
822,615 79.2
1,609,780 28.7
Query/Inference Generation. As no other re- source currently has examples of high-quality com- monsense inferences in our proposed open for- mat, we develop set of few-shot examples of 10 contexts (either handwritten selected from ATOMIC10X from ROCStories (Mostafazadeh al., 2016)) with 10 handwritten commonsense query/inference pairs for each (see Appendix B.2 for more details). These query/inference pairs cover a broad swathe of knowledge, including con- sequences, explanations, reactions, attributes, coun- terfactuals, etc.
Inference
2020
602,154 48.3
847,913
52.5
& Tail
10x
874,417
13.5
695,877
21.0
NOVA
2,030,488 93.2
5,835,099 30.0
2020
1,246,582
875,157 $1.9
Total
10x
6,456,300
812,166 21.1
NOVA
2,178.086
7,224.608 28.0
Table I: Statistics of NOVATOMIC compared to exist- ing CSKG, ATOMIC and ATOMIC10X and indicate the count and percentage of unique entries 3- 3-grams, respectively. Compared to previous CSKGs. NOVATOMIC contains more diverse entries with higher lexical variations. Notably, NOVATOMIC adopts open data format by breaking out from fixed relation types, it contains much more diverse and flexible sets of relations denoted with questions that tie premise and hypothesis together.
For each context in NOVATOMIC generated in the previous step, we randomly select and permule n Uniform(1, 10) of the few-shot examples to provide context after the instructions and then task the model with generating 10 query/inference pairs for the new context. The rationale for this random selection and permutation of the few-shot examples is to mitigate the potential overfitting to specific ordering selection ordering of handwritten examples. To try to support the use case where a desired commonsense query is not known advance, e.g. when user simply want general knowledge for a given context, we also generated half of the commonsense hypotheses without generating query first (prompt in B. B.2). At training time ($2.3), we input a NULL value for the query field. We generated all query/in- ference pairs using default decoding arguments with g.t- 5-turbo-030 for total cost of USD $337.16.
flected by the significantly more diverse 3-grams. In particular, NOVATOMIC breaks out from fixed relation types with open questions connect premise and hypothesis, contains much more diverse and flcxible sets of relations denoted by natural language questions.
It is also of note that, based on estimates from (West al., 2022), the total cost of ATOMIC and ATOMIC10X were approximately USD $40,000 and USD $6,000 respectively, whereas the cost for NOvATOMIC is approximately $400. Though the size of NoVATOMIC is somewhat smaller, the unit cost per example is also signif- icantly lower.
Analysis of Question Types. To delve into what relations are encoded in NOVATOMIC with open questions, we conduct an analysis of question types. Figure shows the top 10 most common ques- tion prefixes, including open-ended question types, such as what and how, and binary yes/no ques- tion types, such as is and will. By grouping WH- questions together (i.c., how, what, why, who, obtain 81.1% of
2.1.2 Analysis
Comparison to Previous CSKGs. Table shows the comparisons of NOVATOMIC to exist- ing CSKGs, 202 (Hwang et al., 2020) and ATOMICI 0x (West et al., 2022) in dataset statis- tics and lexical diversity measures. NOVATOMIC contains more diverse unique premises (heads) and hypotheses (tails) as indicated by the higher num- ber and/or percentage of these data entries. No- VATOMIC also has higher lexical variations, as re-
where, when, whose, which), open-ended questions and 18.9% of binary yes/no questions, indicating a diverse and flexible relation space the large portion of free-form questions repre- sent, as shown in Figure 2(b). Table shows some
httpss//catalog.datt http://aaast//aaayyaaamasss from- oocial-ecuuity-cardee htt.:/uutttyctttppppiiiiiooooona

---

600000
annotator per example. For underlying data, we use 16k examples from NOVATOMIC, as well as 2k ex- amples each from ATOMICI0X and ATOMIC2020 to increase diversity of annotated knowledge style. While these knowledge graphs have fixed relation sets, we use sample natural language queries to replace the discrete relations (e.g. xNeed Whal did PersonX need?).
400000
18.9%
Yes/No
200000
61.1%
Open-ended
what how
why who
will
where are
do
can
Open-ended Yes/No
(a) Question Prefix
Figure 2: (a) The most frequent question prefixes. (b)
We conduct a human agreement study on a seg- ment of 200 examples for which we elicit 3 amnota- tions each, finding Fleiss k (Fleiss, 1971) of 0.317 indicating fair agreement (Landis and Koch, 1977).
The composition of open- open-ended vs. yes/no questions.
Most Frequent Questions
What time is it?
Who is PersonX?
2.3 Training NOVACOMET
What is the weather like?
What is the prerequisite for this situation?
2.3.1
Commonsense field masking
What is the consequence of the situation?
Previous knowledge models tend to use standard reaattooation tail format in training, generating some inference given the situation/concept, and one of a set of possible commonsense relations guide generation.
What is the counterfactual of the situation?
What will happen next?
What is the occasion?
What is the relationship between PersonX and PersonY?
Where they?
Table 2: Frequent queries in NOVATOMIC. Note that take the top 100 surface forms, and cluster them into semantically related/equivalent groups by hand. Queries above represent the top groups by aggregate count, with indicative labels. See Appendix A for more details.
The goal of NOVACOMET is maximum flex- ibility in handling commonsense knowledge and tasks, meaning we would like generate any of these fields from any others. For example, may want to generate a likely query that connects the context and inference; or, a context under which the query and inference are correct. To this end, we propose commonsense field masking, wherein we randomly sample subspans of fields to be masked for prediction, e.g.
of the most common questions in the dataset. The most common questions are not context-specific (asking about time, weather, or location), although we find that many of the queries do condition specif- ically on context.
Input:
Context: Consider the list of MASKg shows. Query: What is the MASKg show?
2.2 Plausibility Annotation
Next, we collect annotations of CQI data plausi- bility. Broadly, this follows West et al. (2022); Howard et al. (2023) in combining generated data with an automatic critic to maximize the quality of final trained model. In this case, however, explore multiple ways to incorporate annotations of plausibility into the final model NOVACOMET ($2.3.2).
Inference: Hamilton
Target:
MASK; Broadway
MASKg most popular
The process of masking follows two steps. First, the set of which fields (CQI) will be masked is uniformly selected from all options in which at least one field is masked. Second, for each field, we randomly (with p=0.5) decide whether to mask the entire field, or subspan. Finally, for those fields where a subspan is masked, we uniformly select the mask length, and which subspan of the given length to mask.
Our primary means collecting annotations of plausibility is through Amazon Mechanical Turk. We use a similar annotation template to (Hwang et al., 2020) (see Appendix C), asking annota- tors to decide if knowledge is always/often, some- times/likely, farreleeeeyffffeehheheeeeee true, or invalid (giv- ing these annotations score of 3, 2, respec- tively). We consider any knowledge scored to be plausible.
In effect, this gives the final model maximal flex- ibility inference time. Users can mask any field, either the full span or infill subspan as needed, allowing for use cases besides simply generating a full inference as in previous commonsense models.
In practice, we collect 20k annotations, with

---

We explore how this can be especially useful in $3.2.
2.3.3 Model Training
We use the T5X codebase (Roberts et al., 2022) to train NOVACOMET, using the base T5 1.1 xxl (~1 1B parameters) checkpoint to initialize all of our experiments. We train all models on v3-128 TPU pods, using a batch size of 128 and learning rate of le-5. For generation models, we train for a fixed 100k training steps, ensuring that loss does not converge begin increasing. For models that include plausibility prediction as an objective, we stop training when evaluation loss for plausibility converges, which is often significantly before 100k training steps.
2.3.2 NOVACOMET Versions
We consider variety of methods to use the genera- tion and critique data described above for training.
Generation-only Model First, we consider the simplest option for producing commonsense gen- eration model: training directly NOvATOMIC. NOVACOMET base is trained only generation data from $2.1 with the commonsense masking objective ($2.3.1). Plausibility not used in this
model.
Experiments
Critic-only Model Second, train stand- alone plausibility critic model, NOVACOMET, crit.
3.1 Evaluating Plausibility
This trained to generate plausibility score from a complete CQI (context, query, inference) knowl-
We begin by evaluating the performance of our plausibility model NOVACOMET critic- Particu- larly, we aim to understand the ability of this model to provide useful, absolute plausibility score. We compare the accuracy of our plausibility scores on discriminative commonsense benchmarks judge its effectiveness.
edge triple, on the annotation set from $2.2. In effect, it returns probability that given CQI is plausible.
Filtered Generation Model Following West et al. (2022), we use simple filtering-based tech- nique for improving generation with plausibility scores. Using NOVACOMET crit, calculate the probability of being plausible for all examples in NOVATOMIC, and filter to only those points that meet a minimum probability. We focus on one threshold in particular, 0.99, indicating that NOVA- COMET crit gives at least 0.99 probability to the given CQI being plausible. We call the resulting model NOVACOMET filter-0.99. and the resulting filtered training set retains over 50% of its origi- nal size, indicating NOVATOMIC is already high quality.
3.1.1 Datasets
We consider range of standard discrimina- tive commonsense benchmarks: HellaSwag (HS) (Zellers et al., 2019) for generation recognition; oNLI (Bhagavatula et al.. 2019) for abductive reasoning; WinoGrande (WG) (Sakaguchi et al., 2019) for pronoun resolution; Commonsense QA (CSQA) (Talmor et al., 2019) and CODAH (Chen et al., 2019) for general commonsense question answering; Social IQA (SIQA) (Sap et al., 2019) for social commonsense; RiddleSense (RS) (Lin et al., 2021) for riddle answering; and Physical IQA (PIQA) (Bisk al., 2019) for physical common- sense. Together, these allow us to judge the ability of models to assess the correctnes//lausiiiity of commonsense.
Quantized Reward Conditioning Inspired by quantized reward conditioning in (Lu al., 2022), we also consider more closely unifying the critical and generation data. We consider a liggt- one-step approach (as opposed to full reinforce- ment learning in Lu et al. 2022) in which we an- notate NOVATOMIC with NovACOMET crit, then train musked-predicttee model that includes plau- sibility as conditioning variable for predicting CQI. For annotation with NOVACOMET.c.. orit. greedily decode plausibility, and train reward- conditional model NOVACOMET When decod- ing with NOVACOMETre. we condition on either the "plausible" labels (2 or 3) from $2.2.
3.1.2 Models and Baselines
As baselines, we primarily consider popular lan- guage models in a roughly similar range of pa- rameter sizes. We include basic language model LLaMA (Touvron et al., 2023) and PaLM (Chowd- hery al., 2022) (citing performance directly for both); and language models with general task tun- ing such as QA for Macaw (Tafjord and Clark, 2021) or instruction tuning for Flan-T5,, (Chung et al., 2022b) and TO (Sanh et al., 2021). We create

---

standard-formal prompts that depend on the model. When possible, models are given answer choices as input. This an advantage over plausibility models like NovACOMET which are designed to judge answers in isolation, but we include this to maximize baseline quality. To score options of baselines, we use negative-log-likele likeeiioooo as it was found by us to be best out of range of options. We cite results for alternative format- ting for prompting FLAN from (Liu al.. 2023) which automatically reformats commonsense ques- tions as statements, then judges plausibility as the likelihood of answering "yes" or "no" to whether the statement is plausible. We note that, while this method performs well, it will not generally apply to Context-Query-Inferee
to improve the overall dataset) we are judging ex- amples in isolation with no clear contrastive ex- amples. The competitive performance of NOVA- COMET here, despite such disadvantages, fur- ther validates it for this use case.
3.2 Evaluating Generation
The central goal of NOVACOMET is in generating commonsense knowledge, and carrying out com- monsense reasoning. In this section, we test the ability of various versions of NOVACOMET de- scribed in $2.3 to do this. Note that primar- ily use human evaluation for model generations. following similar setup $2.2 with annotation templates available in Appendix C.
(CQI) formatted data, as not all CQI examples be naturally reformatted into short statements, but include this baseline for completeness. We also cite results on GPT-3.5, ChatGPT, and GPT-4 from the same paper.
3.2.1 Datasets
First, we test the ability of models to generate com- monsense knowledge in the format of previous knowledge models. Particularly, we take a sam- plc of ATOMIC (Hwang et al., 2020) common- sense prompts (head relation), testing the ability of models to generate valid tail. Results are in- cluded in Table 4.
We compare baselines to NOVACOMET ont de- scribed in $2.3. For this models, we score options based on the probability of predicting 2 or 3 for plausibility (sometines/lkkely or always/often true), renormalized against the probability of predicting or 0 (rarely never true, or invalid).
Next, we test on various downstream bench- marks requiring generative commonsense reason- ing. First, we test abductive natural language gener- ation (aNLG) (Bhagavatula et al., 2019), wherein models must abductively fill in the gap in story between two observations. We also consider two question-anee -answirn-nnswering datasets that require common- sense reasoning: TellMeWhy (Lal et al., 2021) which models explain events, and Reflect (Zhou et al., 2022) in which models generate ATOMIC- style inferences for dialogues. We report results for all downstream reasoning benchmarks in Table 3. We use a custom annotation template for aNLG, and otherwise use the base CQI template from our annotation in $2.2.
3.1.3 Results and Discussion
Model scores on various tasks requiring common- sense knowledge can be seen in Table 3. While various models are better at different tasks, NOVA- COMET crit is tied for most combined 1st 2nd place results (5). Note that the other tied system, Flan-T5 (statements) requires automatically trans- forming each problem into a yes or no question; a transformation that is not generally applicable to the kinds of Context-Query--Ineernnee style prob- lems we would like to solve when deriving com- monsense information from model.
Looking cases where NOVACOMET one fails to get the highest accuracy, perhaps unsurpris- ing that PaLM 540B and 62B outperform all other models on HellaSwag, which requires predicting the most likely continuation of a scene descrip- tion, task especially well suited to a raw language model. Furthermore, with Physical IQA (PIQA), the focus is on physical commonsense, subcat- egory that our base generator seemed to produce less naturally on inspection.
3.2.2 Baselines and Models
For baselines, we include all of the models de- scribed in $3.1 as well as T55 (~11B parameters) finetuned for language modeling (T5-LM) (Raffel et al., 2019). We use straightforward prompts to describe each task and generate directly.
Different datasets can demonstrate unique ways to use the commonsense masking of NOVA- COMET for generation. For example, for oNLG, we mask between the beginning (ol) and ending (o2) events to form a natural sequence:
We also note that many baselines (e.g. Macaw, TO) assume access to all answer choices. For our use case (judging knowledge within NOVATOMIC

---

system
HS
oNLi
CODAH
WG
CSQA
SIQA
CosmosQA
RS
PIQA
Cited Results
GFT-3.5
70.4
76.6
85
72.5
66.9
65.3
84.2
CharGPT GPT-4
43.0
60.3
56.8
61.3
39.6
52.2
67.6
40.0
75.0
66.0
77.0
43.0
57.0
73.0
Plan-T5 (statements)
64.5
80.8
89.6
84.7
69.2
73.2
83,9
Ilama-7B
76.1
70.1
Ilama- 238
79.2
73.0
50.4
80.1
PaLM 62B
79.7
77.0
80.5
PaLM 540B
83.4
81.1
82.3
Comparable General Models
Macaw
50.8
71.6
82.9
60.7
79.4
68.8
70.4
58.8
79.4
Flan-T5,xl
73.5
70.7
58.7
72.9
72.8
55.2
72.9
60.6
82.0
TO
63.7
70.3
73.4
58.9
68.1
66.8
75.4
53.8
84.9
NoVACOMET
74.4
80.4
86.7
79.6
76.7
77.1
800.3
58.6
83.4
Table 3: Comparison of model scores on commonsense benchmarks. Best results are bold and second best are underlined. Note that no other method surpasses NOVACOMET the combined number 1st and 2nd place results (5). Comparison using absolute scores from different models. indicates values cited from (Liu et al., 2023) which uses a pipeline with Flan-T5.u1- 2 indicates values cited from (Chowdhery et al., 2022), 3 indicates values cited from (Touvron al., 2023). Values for large, recent GPT models (GPT-3.5, ChatGPT, GPT4) are cited from (Liu et al., 2023).
oNLG
Reflect valid
TellMeWhy valid
ATOMIÇ valid
system
obs2
obsl
obsl42
overall
Baselines
LLaMA-7B
0.030
0.025
0.022
0.013
0,388
0.463
0.470
LLAMA-13B
0.010
0.008
0.012
0.008
0,456
0.442
0.515
0,260
0,258
0.235
0,248
0.759
0.686
Alpaca-7b
0.162
0.123
0.120
0.122
0,612
Alpuca-13B
0.355
0.290
0.248
0.764
0,660
Flan- -12
0.715
0.627
0.605
0.622
0.618
0,562
0.692
Flan-T3 Man-T5224
0.735
0.653
0.635
0.657
0.796
0.807
0.757
NOVACOMET
NOVACOME'T Meaa hoae
0.877
0.826
0.819
0.814
0.864
0.928
0.847
NOVACOME'T F..e
0.887
0.837
0.837
0.827
0.864
0,916
0.848
NOVACOMETT (EOME)TT)()))
0.837
0.793
0.787
0.797
0,874
0,916
0.861
NoVACOMETTrees)
0.840
0.797
0,780
0.787
0.869
0.918
0.859
Table 4: Human evaluation of various commonsense generation tasks. Note that the basic version of NOVACOMET outperforms baselines consistently, is outperformed by versions that use plausibility to improve. We find human agrecment with Fleiss (Fleiss, 1971) of 0.32, 0.44, 0.43, 0.39 (respective to order in the table) indicating fair to moderate agreement. Note, values in this table are normalized to [0, range.
Input:
inference. For all models, we use greedy decoding.
Context: <o1> MASKE: Query: What happens next?
3.2.3 Results and Discussion
Inference: <o2>
All generation results use human evaluation, pre- sented in Table 4. Note that human evaluation templates are included in the Appendix. We evalu- ate 100 examples for each system and dataset. For Reflect, TellMeWhy, and ATOMIC2020 we use the same template $2.2. For oNLG use tem- plate measuring coherence between the generated infill and either or both hypotheses, as well as over- all quality. All scores in Table 4 are normalized to
To predict hypothesis h that fits between ol and o2. We found this resulted in much higher quality generations than encoding of, o2 as context and predicting à as inference.
For other datasets (Reflect, TellMeWhy, 200, we can encode examples simply by giving context and query, then predicting the

---

range between 0 and 1.
edge distillation from larger models. Following this, several works effectively improved upon this for iterative distillation (Sclar et al., 2022; Bha- gavatula et al., 2023), self-chat with feedback and conversations with ChatGPT (Xu et al., 2023; Geng et al., 2023; Chiang et al., 2023). SODA (Kim et al., 2023) contextualized social commonsense knowl- edge from a knowledge graph to distill dialogues from InstructGPT. Sclar al. (2022) established filters based on length, fidelity, and information bot- tleneck for distilling reference-free summarization determining the effectiveness of designing filters for selecting data for the following iteration. Re- cently, (Jung et al., 2023) proposed framework to learn high-quality model from low-quality teacher model to distill a good dataset by summa- rizing and paraphrasing.
Note that NOVACOMET models win across the board. Particularly effective the filtered model NOVACOMET faiter--oaaar 0.99, but so are the reward conditional models, and NOVACOMET (e(2) in par- ticular, conditioned on "2" (likely/ommetimes true) rather than "3" (always/often true). It is possible that answers that are always true are somewhat less creative preferable humans.
fälter
In general, the NOVACOMET models that use plausibility information outperform the basic No- ALCOMETbase: other than on the TellMeWhy dataset. This demonstrates particular advantage of distilling discrete data - can be annotated, and those annotations can improve downstream perfor- mance.
Overall, superior performance of NOVACOMET suggests that explicitly modeling knowledge can provide an advantage, at least considering tasks that explicitly require commonsense knowledge and reasoning.
Conclusions
Overall, we introduce NOVACOMET, open commonsense foundation model. NOVACOMET takes advantage of closed proprietary models, re- sulting in an open pipeline and resources that are publicly available. NOVACOMET is trained on data generated from these closed proprictary mod- els and augmented with human annotations, result- ing in both a high-quality plausibility model and improved generative model. NOVACOMET sur- passes other general models of similar size at a range of commonsense knowledge-intenss inttensvveteeivveeee tasks, demonstrating the existing need for explicit knowl- edge modeling. even as task-focused methods like instruction tuning grow in popularity.
4 Related Work
Knowledge Generation Pretrained language models demonstrated the ability to carry implicit knowledge (Petroni et al., 2019; Dhingra et al., 2022). These large language models are prompted for generating new knowledge to perform down- stream tasks such as text classification (Shin et al.. 2020; Puri and Catanzaro, 2019), commonsense reasoning (Liu et al., 2022b; Trinh and Le, 2018; Davison et al., 2019). We take inspiration from commonsense LMs, designed for query common- sense knowledge, such as COMET (Bosselut et al., 2019) and COMET-2020 (Hwang et al., 2021). Domain specific LMs are also used for knowl- edge graph completion in specialized domains like biomedicine (Nadkarni et al., 2021). Liu et al. (2022a) use dataset cartography to prime the model with challenging examples and enable it to generate more examples with such patterns.
Limitations
First, we recognize that our line of research re- quires extensive resources and funding, limiting the broad adoption of our methodology it is pre- sented. Particularly, sive generation from proprietary language models (GPT-3 turbo) and extensive use of TPU resources. Our hope that these barriers will only be lowered as proprietary LMs become cheaper and LMs be- come increasingly efficient to tune and do inference on (Dettmers et al., 2023), lowering the barrier for techniques such as ours.
work relies both mas-
Knowledge Distillation As the process of man- ually creating datasets can be costly and complex, prior studies have explored the realm of automated data generation. These prior works mainly focused extractive approaches, c.g. syntactie parsing (Zhang al., 2020a) or pattern matching (Li et al., 2020) from unstructured text (Lehmann et al., 2015; Buck et al., 2014).
Second of all, we recognize that, while we have attempted test the query- of commonsense knowledge via automatic and human evaluations on a number of different tasks [FIX ME] How- ever, current tasks are largely biased towards both
West et al. (2021) proposed filtering out low qual- ity data using a critic model for symbolic knowl-

---

certain topies and tends to implicitly define ground truth from certain, fixed perspectives rather than ac- knowledging the underlying diversity of human per- spectives (Santy et al., 2023). This limits our abil- ity to assess whether our models capture genuine human agreement- or only the agreement of cer- tain portion of the oopulation-somethine which we hope future work will investigate.
Ning Bian, Xianpei Han. Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. 2023. Chatgpt is a knowledgeable but inexperienced solver: An investigation of com- monsense problem large language models. ArXiv, abs/2303.16422
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. PIQA: Reasoning about physical commonsense natural language. AAAI Conference on Artificial Intelligence.
Ethics Statement
Antoine Bosselut, Hannah Rashkin, Maarten Sap. Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. Comet: Commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 4762-4779
Akin to all other machine learning approaches, our model could inadvertently exhibit biases. We ac- knowledge that the open format relations gathered from proprietary models may not be representative of all cultures, and thereby these perpetuate the biases that these proprietary large models possess. While generating commonsense knowledge, LLMs may result in unanticipated commonsense infer- ences, including those that are biased and escape our critic model. Consequently, incorporating these inferences during training can further amplify such biases. We are committed to understanding such biases and improving our critic model. However, our model's 's central tenet is knowledge, which con- trasts with existing public models of similar size and architecture, thereby regulating the toxicity of the model. We ensured that the crowd workers in- volved in our project were compensated at a rate that met or exceeded the minimum wage require- ment, recognizing the value of their contributions to building our model. Comparable to all open models, our model is susceptible to malicious use and it our collective responsibility thrust safe open usage. We acutely understand the ethical im- plications associated with our proposed method and are dedicated to resolving them, aiming to en- sure the responsible adaptation of our approach in the community.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Fric Horvitz, Ece Kamar, Pe- ter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general in- telligence: Early experiments with gpt-4.
Christian Buck, Kenneth Heafield, and Bas Van Ooyen. 2014. N-gram counts and language models from the common crawl. In LREC, volume 2, page 4. Citeseer.
Michael Chen, Mike D' D'Arcy, Alisa Liu, Jared Fer- nandez, and Doug Downey. 2019. Codah: An question answering dataset for common sense. In Proceedings of the 3rd Work- shop on Evaluating Vector Space Representations for NLP, pages 63-69.
adversarially-
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chathot impressing gpt-4 with 90%* chatgpt quality.
References
Chandra Bhagavatula, Jena D. Hwang, Doug Downey, Ronan Le Bras, Ximing Lu, Keisuke Sakaguchi, Swabha Swayamdipta, Peter West, and Yejin Choi. 2023. 12D2: Inductive knowledge distillation with NeuroLogic and self-imitation. In Proceedings of the st Annual Meeting of the Association for Computa- tional Linguisties, ACL.
A Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao. Parker Barnes, Yi Tay, Noam M. Shazeer, Vinod- kumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat,
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Scott Yih, and Yejin Choi. 2019. Abductive commonsense reasoning. ICLR.

---

Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyzz, Erica Moreira, Re- won Child, Oleksandr Polozov, Katherine Lee. Zong- wei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz. Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Hellst-Hel Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. ArXiv, abs/2204.02311.
2023. Koala: A dialogue model for academic re- search. Blog post.
Phillip Howard, Jumlin Wang. Vasudev Lal, Gadi Singer, Yejin Choi, and Swabha Swayamdipta. 2023. Neu- rocomparatives: Neuro-symbolli distillation of com- parative knowledge. ArXiv, 23497797788..
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2020. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. AnAA Conference on Artificial Intelligence.
Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras. Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. (comet-) atomic 2020: On sym- bolic and neural commonsense knowledge graphs, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6384-6392.
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao. Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe Le, and Jason Wei. 2022a. Scaling instruction finetuned language models. ArXiv, abs/2210. 10.1 1416.
Jachun Jung. Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. 2023. Impossible distillation: from low-quality model to high-quality dataset & model for summa- rization and paraphrasing. ArXiv, 0aas/2355.66333
instructionfinetuueed
Hyung Won Chung. Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu. Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022b. Scal- ing cnttructtioniinttueee language models. ArXiv, 1446.
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap. et al. 2023. SODA: Million-scale dialogue distillation with cial commonsense contextualization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP.
Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2021. TellMeWhy: A dataset for answering why-questions in narratives. In Findings of the Association for Com- putational Linguisties: ACL-IJCNLP 2021, pages 596-610. Online. Association for Computational Lin- guistics.
Joe Davison, Joshua Feldman, and Alexander M. Rush. 2019. Commonsense knowledge mining from pre- trained models. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing,
Richard Landis and Gary G Koch. 1977. The mea- surement of observer agreement for categorical data. biometrics, pages 159-174.
2019, Hong Kong, China, November 3-7, 2019, pages 1173-1178. Association for Computational Linguis- tics.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, D. Kontokostas. Pablo N. Mendes, Sebastian Hell- mann, M. Morsey, Patrick van Kleef, S. Auer, and C. Bizer. 2015. Dbpedia- a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6: 6:16
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer 2023. Qlora: Efficient finetuning of quantized lls.. arXiv preprint arXiv:230S.I 14314.
Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, and Benjamin Van Durme. 2020. Guided generation of cause and effect. In Proceedings of the Twenty- Ninth International Joint Conference on Artificial Intelligence, IJCAI-20.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos. Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language mod- els temporal knowledge bases. Trans. Assoc. Com- put. Linguistics, 10:257-273.
Bill Yuchen Lin, Ziyi Wu, Yichi Yang, Dong- Ho Lee, and Xiang Ren. 2021. RiddleSense: Reasoning about riddle questions featuring linguistic creativ- ity and commonsense knowledge. In Findings of the Association for Computational Linguistics: ACL- IJCNPP 2021, pages 1504-1515, Online. Association for Computational Linguistics.
Joseph L Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological bulletin, 76(5):378.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wal- lace, Pieter Abbeel, Sergey Levine, and Dawn Song.

---

Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022a. WANLI: Worker and ai collabo- ration for natural language inference dataset creation. In Conference on Empirical Methods Natural Lan- guage Processing.
Fabio Petroni, Tim Rocktäschel, Patrick S. H. Lewis. Anton Bakhtin, Yuxiang Wu, Alexander H. Miller. and Sebastian Riedel. 2019. Language models knowledge bases? CoRR, 0e//1999000666
Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. CoRR, abs/1912.10165.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Pe- ter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022b. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics ( (Volume| I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 3154- 3169. Association for Computational Linguisties.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/ Abs/1990.1. 10683.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sha- ran Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez, Alex Salcianu, Mare van Zee, Jacob Austin, Se- bastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdh- cry, Jasmijn Bastings, Jannis Bulian, Xavier Gar- cia. Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp. Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma. Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Bren- nan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo, 2022. Scal- ing up models and data with t5x and seqio. arXiv preprint arXiv:2203.1 17189.
Jiacheng Liu, Wenya Wang, Dianzhuo Wang. Noah A. Smith, Yejin Choi, and Hanna Hajishirzi. 2023, Vera: A general-purpose plausibility estimation model for commonsense statements. ArXiv, fbs/2305.03699
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. 2022. QUARK: Controllable text generation with reinforced unlearning. In Advances in Neural Information Processing Systems.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruy Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849, San Diego, California. Association for Computational Linguis- tics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2019. Winogrande: An adver- sarial winograd schema challenge at scale. In AAAI Conference on Artificial Intelligence.
Victor Sanh. Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matto Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Stella Rose Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multi- task prompted training enables zero-shot task gener- alization. ArXiv, abs/21 10.08207.
Rahul Nadkarni, David Wadden, Iz Beltagy, Noah A. Smith, Hannaneh Hajishirzi, and Tom Hope. 2021. Scientific language models for biomedical knowledge base completion: An empirical study, In 3rd Con- ference on Automated Knowledge Base Construction. AKBC 2021, Virtual, October 4-8, 2021.
OpenAL. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Long Ouyang, Jeff Wu. Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke F. Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, abs/2203.021555
Sebastin Santy. Jenny T Liang, Ronan Le Bras, Katha- rina Reinecke, and Maarten Sap. 2023. NlPosition- ality: Characterizing design biases of datasets and models. In Proceedings of the 61 Annual Meet- ing of the Association for Computational Linguisties, ACL.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th annual meeting on association for compu- lational linguistics, pages 311-318. Association for Computational Linguistics.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQA: Common- sense reasoning about social interactions. In Con- ference on Empirical Methods in Natural Language Processing.

---

Melanie Selar, Peter West, Sachin Kumar, Yulia Tsvetkov, and Yejin Choi. 2022. Referee: Reference- free sentence summarization with sharper controlla- bility through symbolic knowledge distillation. arXiv preprinl arXiv:2210. 13800.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can machine really finish your sentence? In Annual Meeting of the Association for Computational Lin- guistics.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with au- tomatically generated prompts. Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pages 4222 4235. Association for Computational Linguistics.
Hongming Zhang, Daniel Khashabi, Y. Song. and D. Roth. 2020a. TransOMCS: From linguistic graphs to commonsense knowledge. In ISCAI.
Tianyi Zhang, Kishore, Felix Wu, Kilian Q. Wein- berger, and Yoav Artzi. 2020b. Bertscore: Evaluating text generation with bert. ArXiv, 0bs/1994.0900
Pei Zhou, Hyundong Justin Cho, Pegah Jandaghi. Dong- Ho Lee, Bill Yuchen Lin, Jay Pujara, and Xiang Ren. 2022. Reflect, not reflex: Inference-based common ground improves dialogue response quality. In Con- ference on Empirical Methods Natural Language Processing.
Oyvind Tafjord and Peter Clark. 2021. General- purpose question-anssweiig with macaw. ArXiv, abs/2109.02593.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseCA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologiex, Volume I (Long and Short Pupers), pages 4149 Minneapolis, Minnesota. Association for Computational Linguistics.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' elieelin Rodriguez. Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.
Trieu Trinh and Quoe 2018. A simple method for commonsense reasoning. CoRR, abs/1806.028..
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 4602-4625, Seat- tle. United States. Association for Computational Linguistics.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2021. Symbolic Knowledge Distillation: from general language mod- els to commonsense models. In North American Chapter of the Association for Computational Lin- guistics.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023. Baize: An open-source chat model with parameter-efficien tuning on self-chat data. ArXiv, abs/2304.01 196.

---

A Manual Cluster Analysis of Queries
Generate 20 everyday events about PersonX (one per line). It may also involve other entities, such as PersonY.
To understand the contents of NOVATOMIC, we analyze the top 100 surface form queries by total count in NOvATOMIC. We cluster these queries by hand into semantically related/equivalent groups, and then further take the top 10 of these groups. displayed in the main paper text. In Table 5, we include all queries in the the top 10 clusters along with with counts and total counts per cluster.
1. Event:
Generate 20 situations.
1. Situation:
A.1 Automatic Evaluation of Generation
Generate 20 common si situations.
We also include automatic evaluation with 2 met- ries in Table We find these values show much less distinct spread, with no model taking clear lead over others. The seemingly lower information and general unreliability of automatic metrics was a motivation in mainly considering human evalua- tion.
1. Situation:
Generate 20 everyday situations.
1. Si Situation:
Generate 20 situations that happen often
B Data Generation
B.1 Context Generation Prompts
1. Situation:
Below are the 21 prompts used for doing context generation (delimited with "))
...
Generate 20 situations that happen sometimes.
Generate 20 events.
1. Event:
1. Situation:
..
Generate 20 situations that include a person or people.
Generate 20 common events.
Event:
1. Situation:
...
Generate 20 everyday events.
Generate 20 everyday situations about PersonX (one per line). It may also involve other entities, such as PersonY.
1. Event:
Generate 20 events that happen often.
1. Situation:
Event:
...
...
Generate 20 situations. They should be complex and include multiple parts. (One per line)
Generate 20 events that happen sometimes
1. Event:
1. Situation:
..
Generate 20 events that include a person people.
Generate 20 common si tuaatinn.. They
should be complex and include multiple parts. (One per line)
1. Event:

---

What time it?
800
What time of day is it?
6153
What is the time of day?
458
What time of the day is it? Total
328
7739
Who PersonX?
2333
What is attrihutc of PersonX?
434
Who are PersonX Person
257
What is PersonX?
233
Who PersonY?
825
Total
4082
What the weather like?
What's the weather like?
382
What is the weather like outside?
351
How is the weather? Total
261
3954
What is the prerequisite for this situation?
640
What is prerequisite for this situation?
517
What is prerequisite for this event?
345
Total
1502
What is the consequence of the situation?
314
What's potential consequence of this situation?
311
What is a potential consequence of this situation?
225
What is the consequence of this situation?
197
What is consequence of this situation?
190
What could be consequence of this situation?
130
Total
1367
What is the counterfactual of the situation?
570
What is the counterfactual of this situation?
202
What is counterfactual of the situation?
163
What is counterfactual of this situation? Total
125
1060
What will happen next?
268
What will the person do next?
211
What will PersonX next?
210
What will they do next?
156
What might happen next?
155
Total
1000
What the occasion?
712
What's occasion?
149
Total
861
What the relationship between PersonX and Person
655
What is their relationship?
198
Total
853
Where are they?
193
Where PersonX?
233
What is the setting?
223
Where this taking place?
135
Total
784
Table 5: The surface forms and counts included in the top 10 clusters of analyzed queries.
Situtioo:
1. Si Situation:
...
...
Generate 20 everyday tituutions. They should be complex and include multiple parts. (One per line)
Generate 20 situations that happen often They should be complex and include multiple parts. (One line)

---

BLEU
BERTScoree TellMcWhy
system
oNLG
Reflect
TellMeWhy
ATOMIC
oNLG
Reflect
ATOMIC 2020
LLaMA-7B
0.8
0.4
2.1
0.1
85.2
83.2
85.9
81.7
LLaMA-13B
1.0
0.5
4.6
0.1
85.4
83.8
86.5
81.7
TO
1.3
3.2
9.0
0.5
87.2
88.7
89.1
85.3
Alpaca-7h
1.3
1.1
6.4
0.3
88.7
87.8
89.4
83.5
Alpaca-1.5 13B
2,4
1.2
6.9
0.2
88.8
87.4
89.2
83.3
Flan-U12
4.3
3.4
5.7
0.5
90.0
86.5
85.9
85.3
Flan-T55pt
4.3
4.4
10.8
0.5
90.0
88.2
90.2
86.3
NOVACOMET
3,4
3.7
10.8
0.6
89.7
88.5
90.8
85.8
Table 6: Comparison of baselines and the NOvACOMET, baee using automatic scores BLEU (Papineni et al., 2002)
and BERTScore (Zhang et al., 2020b). Automatic metrics not seem to agree with human evaluation, and show less clear variation.
Situation:
B.3 With queries
...
Generate 20 situations that happen sometimes. They should be complex and include multiple parts. (One per line)
Given a situation, ask and answer ten
(10) relevant questions that require commonsense or a world model. Some examples may include potential consequences, explanations,
Situation:
prerequisites reactions, attributes, or counterfactuals: The commonsense facts may be about actors, actions, events, or ideas the passage. The examples should be high-quality and things that are true. Please give plausible answer at all times instead of just saying that it depends. Only ask questions that will have relevant, commonsense answer.
Generate 20 situations that include person or people. They should be complex and include multiple parts. (One per line)
Situation:
Generate 20 everyday situaiions about PersonX (one per line). may also involve other entities, such as PersonY. They should be complex and include multiple parts. (One per line)
Alisa and her family lived in Florida.
They heard a hurricane was coming. They decided to evacuate to a relative's s house. They arrived and learned from the news that it terrible storm.
Situation:
B.2
Relation Generation Prompts
What will happen now? They will wait out the storm.
Below are the prompts for generating relations. To promote diversity, the number of examples were randomly selected from Uniform(1, and were shuffled. Some contexts come from ROCStories (Mostafazadeh et al., 2016) and (West et al., 2022), while others are handwritten. All questions and inferences hand-written. When prompting 'gpt- 3.5-turbo', we provide the instructions "Given situation... answer" as the system message, the Context as a user message, and the ten generated queries/inferences as the system response.
2. How does Alisa feel? She is probably relieved to be out of the hurricane' path.
3. what would have happened if Alisa and her family had not evacuated? They would have been in the storm.
4. Why did they decide evacuate relative's house? They wanted to be in safe place.
5. Alisa's family is what? Responsible

---

What might have prevented them from fleeing? If they had not heard about the hurricane, or they had no way to get to a relative's s house.
4. What could hinder this si situation? The woman wants to take the stairs be healthy
5. Is she alone? She may or may not be alone, since there could be other people in the elevator.
7. They would not have fled if they were not what? Cautious
8. Where does their relative live? Somewhere safe from the hurricane.
6. What does the woman see in the elevator? Buttons to different floors What does the woman feel? The woman could feel impatient at having wait for an elevator
9. Should they have fled even if the storm hadn't t been bad? Yes, because they might have not been able leave if the hurricane got worse.
10. How could you describe their relative? Kind
8. As a consequence, what will happen? The woman will arrive at her desired floor
A robber steals from bank.
9. What could prevent this from happening? The elevator is out of service
1. What are potential characters in the situation? Robber, bank teller/ worker, customers
Where are elevators located? Multi- story buildings
Tell me something about the robber?
The robber is probably armed
3. What does the robber have? The robber probably has a getaway car 4. What does the bank teller feel? The bank teller is probably scared
Emma has a big exam tomorrow. She got so stressed, she pulled an all-nighter. She went into class the next day, weary as can be. Her teacher stated that the test is postponed for next week.
5. What might happen to the robber? The robber could go jail
What does the bank have? The bank might have a security system
1. How does Emma feel about this? Emma is probably relieved
7. Before this, did the robber do anything? The robber probably planned this in advance
2. Why might Emma be frustrated? Emma could be frustrated because she stayed up all night studying for nothing
8. As a consequence, what will happen? After, the robber will have the money
3. what is the consequence of the si situation? Emma will have more time study
9. What happens before this? The robber tells the bank teller to give them the money
4. what the prerequisite for this si situation? Emma needed to have a big exam
10. How much money does the robber get?
A lot of money
5. Tell me what Emma will do next. Emma will probably home and sleep.
The woman enters the elevator
6. What did Emma do before this? Emma was studying for her exam
What did the woman have do before?
The woman had to press the button
7. Why did the teacher postpone the The teacher may have postponed the exam because not everyone was ready.
for the elevator to come to her floor
What is the woman's goal? The woman wants to go to a different floor 3. What will the woman do next? The woman will press the button for the floor she wants to go to
8. What is an attribute of Emma? Emma is a procrastinator.
9. What is an attribute of Emma s teacher? flexible

---

10. What is the counterfactual of the si tuation? If Emma didn't have big exam, she wouldn's t have pulled an all-nighter.
7. Where might her phone be? Ivette's phone could be in the house, outside or the car.
8. Did Ivette mean to lose her phone? No
9. What is a consequence of the
Karen was assigned roommate her first year of college. Her roommate asked her to go to nearby city for concert. Karen agreed happily. The show was absolutely exhilarating.
si tuatioon? Ivette will have to buy
new phone
10. What would remedy the si situation? Finding Ivette's phone
What's something we can infer about Karen? Karen likes music
PersonX takes PersonY back to the hospital
What will happen because of this? Karen and her roommate will be better friends
1. Why did PersonX take PersonY back to the hospital? PersonY was not feeling well
3. How might this have been prevented?
2. What happened before this? PersonY was discharged from the hospital
If Karen's roommate shy, she
might have asked Karen go
3. What is PersonX and PersonY's
a concert
relationship to eachother? They are either friends or family.
How old is Karen? Young adult 5. Why did Karen agree happily? Karen
4. What would make this hard? PersonX doesn't 't have a car.
wanted to get to know her roonmate
better, make friends, and enjoy a concert
5. Next, what will happen? PersonY will receive medical care.
6. How did Karen and her roommate get to the concert? By car public transportation
6. What happened before? PersonY asked PersonX to take them to the hospital.
What's potential consequence of this situation? Karen might have fun and meet new people
7. What is PersonX? PersonX is kind
8. What is PersonY? PersonY is sick
9. Where are they? They are in a car
8. How does Karen feel? Karen is pleased
10. What is a result? PersonY will get better
9. What does Karen's roommate think of her? The roommate thinks Karen is cool
Mila and her family lived in Florida.
10. When is the concert? The concert is likely night
They heard hurricane was coming. They decided to evacuate to relative's house. They arrived and learned from the news that it was a terrible storm.
Ivette misplaced her phone at her grandparents.
1. What did Ivette do before this? Ivette was at her grandparents
1. What will happen now? They will wait out the storm.
2. How does Ivette feel? Ivette feels frustrated
2. How does Mila feel? She is probably relieved to be out of the hurricane' path.
What will Ivette do next? Ivette will look for her phone
3. what would have happened if Mila and her family had not evacuated? They would have been in the storm.
What could hinder this situation? If Ivette was more careful
5. Ivette is what? Young
4. Why did they decide to evacuate to relative's house? They wanted to be
6. What would make this situation harder for Ivette? Her phone is turned off.
in safe place.
5. Mila's family is what? Responsible

---

8. What could be true to make PersonX reckless? PersonX crosses when there are lots of cars and no crosswalk What might have prevented them from fleeing? If they had not heard about the hurricane, or they had no way to get to a relative's s house.
4. What is something that could happen? PersonX gets hit by car
5. If this didn't happen, what would happen? this didn't happen, PersonX would not get to where they need to go.
7. They would not have fled if they were not what? Cautious
8. Where does their relative live? Somewhere safe from the hurricane.
6. What actors might be in this si tuation? PersonX, drivers, other pedestrians
9. Should they have fled even if the storm hadn't t been bad? Yes, because they might have not been able leave if the hurricane got worse.
what might PersonX be thinking? PersonX might be thinking that they need to get to the other side of the road.
10. How could you describe their relative? Kind
Alegra coyly smiled at the boy as he walked in.
9. What could be true to make PersonX cautious? PersonX waits carefully for the walk signal and looks both ways.
Why did Alegra smile the boy?
Alegra was interested in him. What will the boy do? The boy will notice Alegra.
10. What do people do before crossing the road? People might look both ways to check for cars.
3. What is Alegra's relationship to the boy? They are strangers.
4. what will happen if Alegra keeps smiling at the boy? The boy might talk to her.
Honor decides whether to bike or walk to school.
If the boy doesn't talk her, how will Alegra feel? Alegra will feel awkward.
1. In what situation would Honor choose walk school? It is raining outside.
6. What is the difference between a coy
2. What is Honor? a student
smile and regular smile? A coy smile is more flirtatious.
3. How could Honor be described? Unsure
4. what would make this improbable? Honor lives very far away from the school.
7. How could Alegra be described? Confident
8. Where is this probably located? In public place
5. Why might Honor choose to bike walk? It is faster.
Alegra is probably what age? A teenager or young adult
6. What will happen Honor can't t make up his mind? Honor will be late for school.
10. what would prevent this from happening? Alegra is scared to put herself out there
7. What is a possible reason for why Honor can t decide? He is feeling lazier today.
PersonX crosses the road What is PersonX? A pedestrian What could prevent this from happening? This could be prevented if there was no crosswalk.
8. Either way, Honor will what? Get exercise
9. what is the difference between biking and walking? Biking is faster but requires more effort
3. What is a prerequisite for this event ? A prerequisite for this event is that PersonX wants to cross the road.
10. what is the weather? It might be sunny.
B.4 Without queries

---

List ten (10) commnnsense facts about each situation. Some examples may
terrible storm.
1. They may have left valuables behind
include potential consequences, explanations, prerequisites or reactions. The commonsense facts may be about actors, actions, events, or ideas in the passage. The outputs
2. They may come back to a destroyed house
3. They were smart to evacuate
4. If they didn' t evacuate, they might have died
could also include counterfactuals things that could hinder the event from happening. The examples
5. The hurricane was very bad
6. Now, they will wait out the storm
7. They went to their relatives house because they wanted be in safe place
should be high-quality and things that are true.
8. They wouldn't t have fled if they had not heard about the hurricane
PersonX crosses the road
1. PersonX is probably going to the other side
9. The relative lives somewhere safe from the hurricane
2. Cars are the road Before this can happen, PersonX looks both ways to make sure it's safe PersonX probably has a destination
10. Their relative is kind for letting them stay over
Fatima was assigned roommate her first year of college. Her roommate asked her to go to a nearby city for concert. Fatima agreed happily. The show was absolutely exhilarating.
5. PersonX is probably walking
6. This wouldn't happen if there wasn't
a crosswalk
7. After, PersonX will be on the other
side
1. Fatima has roommat
If PersonX is jaywalking, they might
2. Fatima likes music
get hit by car PersonX might use a crosswalk signal
3. As result, Fatima and her roommate will better friends
10. This couldn't happen if the person wasn t near a road
4. Fatima enjoyed the concert
5. In the future, Fatima may want to go to more concerts
A robber steals from bank.
6. Fatima may be more likely to spend time with her roommate
1. The robber is probably armed
2. The robber probably has getaway car
7. Fatima's roommate is considerate
The bank teller is probably scared This is illegal
8. Fatima's roommate is probably also student
The robber could go to jail The bank might have a security system
9. The roommate thinks that Fatima is cool
7. The robber probably planned this in advance
10. They got to the concert using a car or public transportation
8. After, the robber will have the money
9. Before this happens, the robber tells the bank teller to give them the money
Loretta misplaced her phone at her grandparents.
1. As result, Loretta may be stressed.
10. The robber might wear a mask
2. Loretta may have buy a new phone.
3. This event may have ruined Loretta's weekend.
Addilyn and her family lived in Florida.
They heard a hurricane was coming. They decided to evacuate to relative's s house. They arrived and learned from the news that it
4. This wouldn't happen if Loretta was more careful.
5.. Now, Loretta will probably look for her phone.

---

6. It will be expensive replace her phone if it is lost.
were out of service
8. The woman would have done this if she wanted to take the stairs to healthy
Loretta is young.
8. This situation would be worse Loretta's s phone was turned off.
9. The woman may have been in a hurry
9. Things could be better if Loretta finds her phone
10. She is in multi-story building
10. Loretta did not mean to lose her phone
PersonX takes PersonY back to the
hospital.
1. PersonY has been the hospital before
SAN FRANCISCO Charlotte's husband,
Maxwell, was violently assaulted by
2. The goal of PersonX was to help
a man who brake into the couple's home in San Francisco early Friday morning, the police said. The authorities identified the suspect Lozen, 42, and said they were investigating possible motive. Lozen could be mentally ill Maxwell was likely asleep when the attack happened
PersonY
3. Before this can happen, PersonY must ask PersonX to take them to the hospital
4. PersonY hopes to get medical care
5. PersonY may have been injured before this
6. This couldn't happen if PersonX does not have a car
3. Lozen is either in custody or being searched for by the police
7. PersonY is sick in some way
8. Going to the hospital may be expensive
4. The breaking and entering was likely planned
9. This probably wouldn't happen if PersonY wasn't 't sick
Charlotte was probably not attacked
6. This would have been frightening experience for Charlotte and Maxwell
10. PersonX cares about PersonY
7. If Lozen is caught, he will likely go to jail
Michaela has a big exam tomorrow. She got so stressed, she pulled an all- nighter. She went into class the next day, weary as can be. Her teacher stated that the test is postponed for next week.
8. Lozen's motive might have been personal
9. This wouldn't t have happened if Lozen were not violent
10. Home invasions are usually tated
1. Michaela is relieved that she doesn's have to take the test today
2. Michaela is sad because she worked hard to prepare and in the end didn' have to
The woman enters the elevator
1. Before, the woman pushed the button for the elevator
3. When Michaela stayed up all night she was studying
2. The woman is going to different floor
4. The teacher probably postponed the exam because not everyone was ready.
After, the woman will push the button for her floor Then, she will press the button for her desired floor
5. Next week, Michaela will have to study again
6. Michaela may do better on the exam next week because she will have more time to prepare
5. First, the woman will wait for other people to walk out of the elevator
6. The woman might have been impatient if she had to wait for long time
7. If the exam was today, Michaela would have done poorly
7. This couldn't happen if the elevator
8. The test is in subject that

---

Michaela is struggling in Michaela is procrastinator 10. Michaela hadn't stayed up all night, she would not tired
Keanu decides whether to bike or walk to school.
1. Keanu might not choose to bike if it' raining outside
Keanu is student They are unsure
4. If Keanu doesn't t make up their mind, Keanu will be late for school
5. This is because Keanu is feeling lazy today
6. Either way, Keanu will get exercise
7. Biking is faster than walking but requires more effort
This wouldn't happen if Keanu were more decisive
9. This would be hard if Keanu lived very far away from the school
10. Keanu might choose to bike if it's nice day outside
C MTurk Templates

---

Instructions (cick expandicollspe))
Thanks for perticipeiing. this HiT!
Evaluate the Al's guess. Tell us given the osservation pair, how good the A''s guess on severa dimensions.
note that you mi ght get
observetio pairs multiple each, you
guess, please
guess carefully.
IMPORTANT:
In this new detasel, some of the guesses may be exect or near copies of one the observel lons.
This automate bad, Please respond with Strongty disagres for all questions.
Please be forghving of minor spelling or grammar errons as that's not what test.
Dxamples
Observations
Observation 1;
Mobs11
Oiservation
What happened in between the observation?
(1) @vafuate At's guess.
(1.1) At's quess a sensical and coherent fallowuu event to Observasion stancias
struraly modiraally
modarately
stromlly
unexplained information gaps.
explonatory pescoding Roent Observation 2 leaves no large unexplained information
moongy disggoee
moderann or weaddy dsagges
modersttly weakly mmnggyaggee aares
ssamcias
guess sensical
sly dhaane
moderaaaly coalds desages
modaaalaly meatly
stroraly AUW
are looked at together. It leaves no arge unevo ained information
ssamciss
Say we were to string the sentences a short onecdote...
"Sott Sohop Heteay
Story/flww understoodoble
By wyy.daaee disagree
moderaally coeidy
mandeeatlyy muatyy
strrrply AEoW
fluenn Engiish speaker).
diengree
(Ciptiora) Please anything unclear. you experienced you fedback
Submit
Figure 3: MTurk template for NLG.

---

MouG NG. Tho connan adult content Worar can wion atriee))
park
Alvagoor sptia
in five peogro liiely epention mukio
mocett)
flyou nee in pardoutarr" for Alvese asens Phrass consac:
Sumainno uxttain simple nopurdded by retigng
repart projuctoed ineppropinte langange
Froline offoroleccannn (MSFW Recaedmmnnt ext
Poqpakk repionnue doreg that Almssoo pagple
phone
coniireccconn danguuutyy cisuully ioralcalll mor..
nousout susking poopicinus uae
leve are parios cenci onltorndesss thersggee
seouppeeyy snen mocenss
couple NOTTS:
uas Norghving of getting promectos aner
the armo are obsuucce you cerroiooo the outh of the fect the olyour to dkay rurt tooo unfane fartz
Eackecboktioe Coold Gouudaaa papllg
New ofte dhe the surecis balmme??
Sourryy)
th suecis mrm?
Weeyehhe insedmestee May neterdedhwerr (ver moumeillrooaa
themindis laVine?
Sipaosut Place thithin drom
Figure 4: MTUkk template for CQL


Reviews:
[{"review_id": "7hnkSREI9I", "reviewer": "Reviewer_eW5y", "summary": "", "questions": "N/A", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for finding our work “well motivated and very helpful for some applications” and our paper “well written and easy to follow” \nWe address all concerns below, especially highlighting **new experiments to combine the critic+generation model** into a single model, thus handling a primary concern of the reviewer, that two models are needed. We also carried out a **topic analysis** of open-format queries to be added to the final paper. \n\n**“[experiments] need two (generation and critic) models”, “discriminative tasks … [use] the critic model”**\n\nTo address this concern, that two models are needed for the given experiments (critic + generator), we have trained a **new model** combining both the generation and critic objectives with multi-task training. We **include initial results below** (Tables A and B), showing that combining the models into one general commonsense model **does not significantly affect performance.** Thus, we can release a single **general commonsense NovaCOMET model** with the capabilities of both original models. \n\n\n**“main contribution of NovaCOMET is the support of open format …  authors do not conduct much analysis and evaluation on this part”** \n\nWe agree that further analysis will enrich the paper, and will include more insights about the open format of NovaCOMET and NovATOMIC in the added page for the final version. For instance, we have carried out a **topic analysis of NovATOMIC** on the open-format queries using BERTopic. We find novel query topics that extend beyond ATOMIC, e.g. asking about the time of the given context (example topic query: “what time of day is it?”). We also find topics that cover existing ATOMIC relations, e.g. the xReact relation (example topic query: “what emotions is the person feeling”). \n\nWe will include further details in the final paper version. This analysis uses 10K examples from NovATOMIC, but this will also be expanded.\n\nIn terms of evaluating the open format of the model NovaCOMET, we note that open format is a key aspect of generative evaluation: neither the TellMeWhy nor aNLG tasks fit existing ATOMIC relations, so this evaluation explicitly demonstrates the flexibility of the open-format approach compared to previous commonsense models. \n\n\n**“dataset and model are constructed and trained with existing methods”**\n\nWe agree with the reviewer’s observation that one “main contribution is the collection and release of the dataset and the model”, and also note that this is a new style of open commonsense model, and thus analysis/testing of such a model is also a proposed contribution. This style of symbolic distillation is a relatively new method, and understanding it requires extensive examples and resources which make use of it.\n\n**“counterexample for instruction tuning models. However, the experiments are only done on commonsense tasks”**\n\nWe clarify that the focus of our work is specifically solving commonsense tasks, and NovaCOMET should be seen as a counterexample to instruction tuning in this domain. Instruction models have been among the most effective options for such tasks recently, which is why they are our main point of comparison. We will clarify our explicit focus on the commonsense domain in the final version of the paper. \n\n\n\n**Tables**\n\n| model    |   aNLI |   CODAH |   Cosmos |   CSQA |   CSQA2 |   HS |   PIQA |   RS |   SIQA |   WG |\n|:---------|-------:|--------:|---------:|-------:|--------:|-----:|-------:|-----:|-------:|-----:|\n| critic   |   0.8  |    0.87 |     0.89 |   0.77 |     0.5 | 0.74 |   0.83 | 0.59 |   0.77 | 0.8  |\n| combined |   0.82 |    0.86 |     0.89 |   0.76 |     0.5 | 0.74 |   0.84 | 0.56 |   0.78 | 0.78 |\n**Table A:** Comparing performance of the new, combined NovaCOMET model to our existing NovaCOMET critic model on discriminative tasks. The combined model does not show a significant decrease in performance, with very similar average accuracies (0.784 vs 0.781)\n\n| model    |   aNLG o2 |   aNLG o1 |   aNLG o1+o2 |   aNLG all |   Reflect |   TMW |   ATOMIC |\n|:---------|:----------|:----------|:-------------|:-----------|:----------|:------|:---------|\n| gen-only |      0.74 |      0.82 |         0.72 |       0.72 |      0.87 |  0.86 |     0.86 |\n| combined |      0.73 |      0.82 |         0.71 |       0.72 |      0.86 |  0.84 |     0.84 |\n**Table B:** Comparing the existing generation-focused NovaCOMET model to the new, combined NovaCOMET model on generation tasks. The combined model is able to achieve very similar generation results by human eval to the generation-only model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is in general well written and easy to follow.\n2. The idea of training a open format commonsense knowledge model is well motivated and can be very helpful for some applications.", "reasons_to_reject": "1. The technical novelty is limited because the dataset and model are constructed and trained with existing methods. This is not a big issue though because the main contribution is the collection and release of the dataset and the model.\n2. While the authors claim that NovaCOMET and solve general tasks while also serve and an open format commonsense knowledge model, the experiments on discriminative tasks are using the critic model while it is the generation model that be used as a commonsense knowledge model. The authors also claim the model can serve as a counterexample for instruction tuning models. However, the experiments are only done on commonsense tasks and need two (generation and critic) models. This does not supports the claim.\n3. One of the main contribution of NovaCOMET is the support of open format. However, the authors do not conduct much analysis and evaluation on this part. This severely limits the completeness of the work.\n", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "This paper introduces NovaCOMET, a new commonsense knowledge model that allows open format relations and enables direct application to reasoning tasks. The authors first collect NovaATOMIC by symbolic distillation with GPT3 and open format relations. Then the authors fine-tunes a T5 model on the generated commonsense knowledge data using a open format objective. The authors pre-train both generative and discriminative models. Results show that the discriminative model achieves competitive results compared to other comparable models; and the generative model archives better results on commonsense generation tasks compared to other comparable general models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "oWSvLPwjIP", "reviewer": "Reviewer_4pDd", "summary": "", "questions": "1. It is good to see the data analysis in Section 2.1.2 and the distribution of the type of questions the data covers. Can you show if the open-form questions are enough to cover the relations in previous datasets/models like COMET?\n\n2. Section 2.3.1 introduces an interesting masking strategy that focuses on entities. What is the benefit of this compared to simpler masking strategies or next-word prediction tasks?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for an extensive review, including finding our “contributions are clear and significant”.\nWe believe all concerns can be fully addressed in this rebuttal, and by updating the paper. We particularly note that **we include requested automatic evaluation scores** and will add this evaluation to the final version.\n\n\n**“only human annotation scores are provided on a subset of 100 examples” … “include some automated scores on the entire evaluation sets”**\n\nWe are glad to follow this suggestion! We have **added automatic evaluation,** and **include tables of sample results below** for 2 metrics: BLEU (using SacreBLEU) and BERTScore. Broadly, NovaCOMET is on par with or exceeding baselines, as in the human evaluation. \n\nWe will include a more extensive set of metrics in the paper, e.g. BLEU, ROUGE, METEOR, BERTScore, BLEUERT, and MoverScore.\nWe will also extend human evaluation to 200 examples.\n\n**“did not discuss the limitations of representing commonsense knowledge in the newly proposed context, query, inference format”**\n\nWe agree with the importance of these limitations, and plan to incorporate a full section on this using the extra page in the camera ready. For example, while context-query-inference format allows more flexible application, it also allows more misuses. Constrained relation sets of previous commonsense models made exploitation/misuse difficult by forcing only certain generation functions, while open-format NovaCOMET does not have this intrinsic safeguard.\n\n\n**“compared their approach to … general LMs that were not specifically trained for commonsense reasoning tasks”**\n\nWhile we agree that general LMs are not trained for commonsense tasks specifically, we argue that NovaCOMET is not explicitly trained for most commonsense tasks either. Rather, it is tuned to focus on commonsense knowledge, but has not been trained on e.g. aNLG data before testing on it. Further, general LMs have become some of the strongest models for commonsense tasks, and so serve as a natural comparison point. \n\nWe will add commonsense models (e.g. COMET-distil, COMET-2020) where applicable as further baselines, although early experiments suggest significantly weaker performance than general LM baselines.\n\n\n**“Can you show if the open-form questions are enough to cover the relations in previous datasets/models like COMET?”**\n\nOne piece of evidence in favor of this is the “ATOMIC” column in table 4 of the paper, which tests the ability of models to generate on COMET/ATOMIC data. NovaCOMET models exceed baselines, and we are happy to add previous COMET models as further baselines for this experiment. This demonstrates a strong ability of open-format queries to cover previous relations. \n\n\n**“What is the benefit of this compared to simpler masking strategies or next-word prediction tasks?”**\n\nOur masking strategy is specifically designed for generation use-cases, allowing partial or full masking of any of the 3 text fields (context, query, inference). In comparison, simpler masking strategies (e.g. basic T5 masking) would not account for field structure, or allow for the same variance in generation length as our strategy, which can mask anything from 1 word to a full field. \n\nNext-word prediction tends to offer strong generative performance, but without the flexible use-cases of our technique, for instance generating the most likely query (middle field) while holding the context and inference constant. \n\n\n**Typos**\n\nThank you for pointing out typos! We will fix these in the final paper version. \n\n**Tables**\n\n| model      |   anlg |   reflect |   tmw |   atomic |\n|:-----------|-------:|----------:|------:|---------:|\n| T0         |    1.3 |       3.2 |   9   |      0.5 |\n| Flan-T5    |    4.3 |       4.4 |  10.8 |      0.5 |\n| Flan-UL2   |    4.3 |       3.4 |   5.7 |      0.5 |\n| LLaMA-7B   |    0.8 |       0.4 |   2.1 |      0.1 |\n| LLaMA-13B  |    1   |       0.5 |   4.6 |      0.1 |\n| Alpaca-7B  |    1.3 |       1.1 |   6.4 |      0.3 |\n| Alpaca-13B |    2.4 |       1.2 |   6.9 |      0.2 |\n| NovaCOMET  |    3.4 |       3.7 |  10.8 |      0.6 |\n**Table A:** Automatic evaluation with SacreBLEU score (huggingface). NovaCOMET achieves the top score on 2 of 4 datasets (TellMeWhy and ATOMIC-2020) and second highest on Reflect. \n\n| model      |   anlg |   reflect |   tmw |   atomic |\n|:-----------|-------:|----------:|------:|---------:|\n| T0         |   87.2 |      88.7 |  89.1 |     85.3 |\n| Flan-T5    |   90   |      88.2 |  90.2 |     86.3 |\n| Flan-UL2   |   90   |      86.5 |  85.9 |     85.3 |\n| LLaMA-7B   |   85.2 |      83.2 |  85.9 |     81.7 |\n| LLaMA-13B  |   85.4 |      83.8 |  86.5 |     81.7 |\n| Alpaca-7B  |   88.7 |      87.8 |  89.4 |     83.5 |\n| Alpaca-13B |   88.8 |      87.4 |  89.2 |     83.3 |\n| NovaCOMET  |   89.7 |      88.5 |  90.8 |     85.8 |\n**Table B:** Automatic evaluation using BERTScore. Similar to human evaluation, NovaCOMET seems to be on par with, or superior to most baseline methods, including achieving the top score on the TellMeWhy dataset, and second-highest score on Reflect and ATOMIC-2020. BertScores are significantly more clustered than human evaluation scores.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The contributions are clear and significant.\n2. The paper provides a detailed analysis of the data distribution, which adds to the credibility of the proposed dataset. \n3. The experiments are thoroughly described and the models are evaluated on multiple relevant datasets and compared to reasonable baselines. \n4. The writing is clear and easy to understand.", "reasons_to_reject": "1. Despite that the writing is very clear, some evaluation steps are not fully replicable. For example, in the evaluation of the generation model, only human annotation scores are provided on a subset of 100 examples from each dataset, which may limit the reproducibility of the results. It would be good to include some automated scores on the entire evaluation sets.\n2. The authors did not discuss the limitations of representing commonsense knowledge in the newly proposed context, query, inference format.\n3. The baselines the authors have compared their approach to are all general LMs that were not specifically trained for commonsense reasoning tasks. ", "questions_to_authors": "1. It is good to see the data analysis in Section 2.1.2 and the distribution of the type of questions the data covers. Can you show if the open-form questions are enough to cover the relations in previous datasets/models like COMET?\n\n2. Section 2.3.1 introduces an interesting masking strategy that focuses on entities. What is the benefit of this compared to simpler masking strategies or next-word prediction tasks?", "paper_topic_and_main_contributions": "This paper proposes to use an LLM (GPT3.5) to generate synthetic data for commonsense reasoning. The proposed dataset, NOVATOMIC, is designed in the format of context, query, inference, which overcomes the limitations of fixed relation sets in previous models. The authors demonstrate that their model, NOVACOMET, which is a finetuned T5 on the NOVATOMIC data with a commonsense field masking objective, outperforms strong vanilla LM baselines on multiple relevant datasets.\nContributions: 1. A new synthetic dataset NOVATOMIC for common sense reasoning. The quality of the data seems good, with ample plausibility annotation from crowdsourcing, and the authors promised to open-source the data. The design of the data in the (context, query, inference) format is interesting and supports open-format relations.\n2. Training a plausibility model to filter the data is a good idea.\n3. The authors showed that the data generation pipeline and the finetuning strategy are successful to distill commonsense knowledge from LLM.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 60, the period before \"Next\".\n\nLine 438 and line 450, I think you mean Table 4"}, "extra_text": {}}, {"review_id": "L6bKzclqi3", "reviewer": "Reviewer_a9Hp", "summary": "", "questions": "My questions and suggestions are listed above in reasons to reject. I am open to hearing the authors' rebuttal to my concerns and would be willing to reconsider my evaluation if necessary.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer’s careful attention to our work, finding our contribution “highly valuable for future research” as well as “high-performing and easy to deploy”. We also thank the reviewer for their stated willingness to reconsider their evaluation.\n\nWe believe that reviewer concerns can be fully addressed here and in the final version. In particular, we will report expanded baseline results as requested by the reviewer, with a large portion included in the table below, and the full set to be included in the final paper (we are currently time-limited by GPU resources and OpenAI quota). \n\n**“missing blanks in Cited results in Table 3 is necessary, as the current comparisons seem awkward, with most results left blank.”**\n\nWe include a sample of these new results in the table below (for Llama models on a subset of tasks). Full results are currently running but will require another week to complete. These will be included in the final paper version.\n\n**“include comparisons with GPT3.5, ChatGPT, and even GPT4”**\nThank you for this suggestion! We will include these comparisons in the final paper. We include some discriminative comparisons in the table below, cited from previous work, but will run full discriminative and generative evaluation for these 3 models in the final version, once our OpenAI quota/budget allows this.\n\n\n**“authors have simply reproduced the symbolic knowledge distillation (ATOMIC10X) pipeline with a more advanced LLM (davinci003)”**\nAs the reviewer points out, our work uses Symbolic Knowledge Distillation to allow an open relation set (new in commonsense models) while ATOMIC10X only used existing relations. As well:\n\n- Unlike ATOMIC10X, our critic model is applicable to discriminative commonsense tasks in general as it is not constrained by closed relations (see also response to Reviewer eW5y for a combined generator-critic model)\n\n- Resulting NovaCOMET can be applied to general-format commonsense generation tasks, unlike past commonsense work in this vein\n\n- Our masking strategy results in diverse use-cases, both filling in full fields (context, query, inference), or parts of multiple fields.\n\n\n**“seems to be applying T5's baseline training style”**\nWe would like to clarify that our objective differs from the T5 baseline style, specifically in explicitly targeting use-cases of our context-query-inference data format. It allows for masking of anywhere from single words to a full field (context, query, inference), and explicitly accounts for field-structure by not masking across boundaries. In contrast, the T5 objective simply samples random subspans to infill with a set distribution, not accounting for generation use-cases or structure of the underlying text.\n\n**“loss of formalization, in exchange for diversity and flexibility, may make it challenging to structure and raise more challenges to apply to downstream tasks”**\nWhile we agree that the data/model no longer operate in a fully structured space, they can still be applied to the existing static relation set of ATOMIC. This is clear from the ATOMIC column in Table 4: NovaCOMET is still very effective at previous ATOMIC relations, while the flexibility allows application to extended downstream tasks (e.g. aNLG).\n\nNovATOMIC is indeed less structured than previous versions of ATOMIC. We plan to continue expanding NovATOMIC through generation, and argue that as it becomes more dense, NovATOMIC will contain larger structured subgraphs following the previous ATOMIC relations, as these are a strict subset of our open-format queries and denote natural and common queries.\n\n**“manually expanding the relations in ATOMIC10X to accommodate the prompts in NovATOMIC, they may achieve the same objective.”**\nWe find that the queries (effective relations) in NovATOMIC are quite varied (800k unique queries, from Table 1 in the paper) and so it may be difficult to capture all of these with a static set of relations. However, we agree that expanded relations would narrow this gap, and as we note in our response above, some subset of NovATOMIC likely corresponds (with varying degrees of strictness) to existing relations. We will include more analysis on the semantic variety of queries in the final paper.\n\n\n**Improvements to structure and new citations**\nWe thank the reviewer for the suggestions! We agree that the changes in format will greatly increase readability of the paper, and will incorporate suggested improvements into the final version. We will also include all suggested citations.\n\n**Tables**\n\n\n| model     |   HS |   aNLI |   CODAH |   WG |   CSQA |   SIQA |   PIQA |\n|:----------|-----:|-------:|--------:|-----:|-------:|-------:|-------:|\n| GPT-3.5   | 70.4 |   76.6 |    85   | 72.5 |   66.9 |   65.3 |   84.2 |\n| chatGPT   | 43   |   60.3 |    56.8 | 61.3 |   39.6 |   52.2 |   67.6 |\n| GPT-4     | 40   |   75   |    66   | 77   |   43   |   57   |   73   |\n|:--\n| Llama-7B  | 76.1 |   62.7 |    40.6 | 70.1 |   43.8 |   48.9 |   79.8 |\n| Llama-13B | 79.2 |   62.9 |    42.5 | 73   |   41.2 |   50.4 |   80.1 |\n|:--\n| NC (us)   | 74.4 |   80.4 |    86.7 | 79.6 |   76.7 |   77.1 |   83.4 |\nTable: Sample of extended results to be added. GPT-3.5, chatGPT, and GPT-4 results are currently cited from Liu et al. 2023 but we will run these ourselves once OpenAI quota allows, along with full generative evaluation. Llama values include new experimental results to fill in Table 3 as suggested, as well as those cited from Chowdhery et al. 2022. We will fill in all missing columns for cited models once GPU resources allow this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents two valuable contributions, NovATOMIC and NovaCOMET. NovATOMIC is a commonsense knowledge base that is constructed by distilling LLM with an open relation set and strict quality critic filtering. This approach achieves both diversity and high quality, resulting in a rich and expansive knowledge base. NovaCOMET is a generative model trained upon NovATOMIC that is both high-performing and easy to deploy. These resources will be highly valuable for future research.\n\nFurthermore, the authors' innovative approach to designing their prompting pipeline is noteworthy. For instance, they introduce the context, query, and inference (CQI) paradigm for commonsense knowledge formulation, as well as the commonsense field masking technique, which enhances the generative model's robustness during training.\n\nThe empirical results demonstrate NovaCOMET's high performance on various tasks, confirming the quality of its data source NovATOMIC.", "reasons_to_reject": "While the paper presents solid work, I feel that it lacks novelty. It appears to me that the authors have simply reproduced the symbolic knowledge distillation (ATOMIC10X) pipeline with a more advanced LLM (davinci003). Aside from the resulting resources, the paper's novel contributions are twofold: the construction of an open relation set and masked commonsense generation.\n\nRegarding the first contribution, while the open relation set removes a traditional constraint of commonsense knowledge bases, it also raises the difficulty of using NovATOMIC. There will no longer be systematic relations, only queries, and contexts. This loss of formalization, in exchange for diversity and flexibility, may make it challenging to structure and raise more challenges to apply to downstream tasks. I’m also thinking about the feasibility of manually expanding the relations in ATOMIC10X to accommodate the prompts in NovATOMIC, they may achieve the same objective.\n\nMasked commonsense modeling seems to be applying T5's baseline training style, which is not a novel paradigm, just a different method for training a generative model. It is a countermeasure to deal with the loss of relations in NovATOMIC.\n\nFor the experiments, it would be beneficial to include comparisons with GPT3.5, ChatGPT, and even GPT4 if the authors have sufficient funding for access. This will make the paper more outstanding, even if NovaCOMET fails to outperform them. Additionally, filling in the missing blanks in Cited results in Table 3 is necessary, as the current comparisons seem awkward, with most results left blank.\n\nThe paper's structure and writing could also be improved. While following the writing pipeline of ATOMIC10X is acceptable, it can also be messy. Consider separating NovATOMIC and NovaCOMET into two sections and adding briefings at the start of each main section to inform readers what they will be reading. Some titles, such as NovaCOMET Versions, could be improved by changing them to training strategies. It is also better to state NovATOMIC in the title of respective sections.", "questions_to_authors": "My questions and suggestions are listed above in reasons to reject. I am open to hearing the authors' rebuttal to my concerns and would be willing to reconsider my evaluation if necessary.", "paper_topic_and_main_contributions": "This paper introduces NovaCOMET, an innovative open commonsense foundation model that combines the benefits of knowledge models and general task models. The authors employ a robust pipeline to extract knowledge from expensive, closed large language models and implement stringent quality controls to create an open commonsense knowledge base called NovATOMIC. Subsequently, they use NovATOMIC to train NovaCOMET, an open commonsense foundation model, using a masked commonsense modeling objective. Empirical findings demonstrate that NovaCOMET outperforms other models in both commonsense question-answering and generation tasks.\n\nThe key contributions of this paper are the creation of two new resources: NovATOMIC, an open commonsense knowledge base, and NovaCOMET, a pre-trained commonsense foundation model.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "See reasons to reject above."}, "extra_text": {"Missing_References": "These papers are also relevant to this work:\n\nWang, J., Qu, J., Liang, Y., Li, Z., Liu, A., Liu, G., & Zheng, X. (2023). Snowman: A Million-scale Chinese Commonsense Knowledge Graph Distilled from Foundation Model. arXiv preprint arXiv:2306.10241.\n\nBhagavatula, C., Hwang, J. D., Downey, D., Bras, R. L., Lu, X., Sakaguchi, K., ... & Choi, Y. (2022). I2d2: Inductive knowledge distillation with neurologic and self-imitation. arXiv preprint arXiv:2212.09246.\n\nWang, C., Li, J., Chen, Y., Liu, K., & Zhao, J. (2022, December). CN-AutoMIC: Distilling Chinese Commonsense Knowledge from Pretrained Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 9253-9265).\n\nKim, H., Hessel, J., Jiang, L., Lu, X., Yu, Y., Zhou, P., ... & Choi, Y. (2022). Soda: Million-scale dialogue distillation with social commonsense contextualization. arXiv preprint arXiv:2212.10465."}}]

Author rebuttal:

Meta-review: Reviewers almost unanimously agree that the purpose of the paper is quite clear. The work is quite well-motivated. Some analyses were missing in the manuscript, such as comparison with GPT-4, ChatGPT, etc, but were later added to the rebuttal. However, everyone also unanimously agree that, while the produce of the paper is important and useful, the core methodology is nothing new--except open-format prompting--, which hurts the overall scientific contribution.

# Pros
1. NovaComet significantly outperforms the baseline models (which include many recent proprietary LLMs) on benchmark commonsense QA tasks by significant margin.
2. The open-format masking strategy is interesting and has lead to rich synthetic data, as verified by some reviewers.

# Cons (and suggestions)
1. Reviewer eW5y points out that the paper lacks analysis on the open-format prompting, despite it being its main scientific contribution. The authors should report on the coverage of ATOMIC relations, as promised.
2. The authors should report the combined results with the critic model.
3. The core methodology of generating synthetic data from LLM and finetuning another model is not new at this point.
Decision: Accept-Findings