CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering
Authors: Weiqi Wang
Tianqing Fang
Wenxuan Ding
Baixuan Xu
Xin Liu
Yangqiu Song
Antoine Bosselut
Venue: EMNLP 2023 Findings (2023)
Keywords: commonsense reasoning
conceptualization
zero shot
question answering
TL;DR: We propose CAR, a zero-shot commonsense question-answering framework that leverages conceptualization. Experiments demonstrate that CAR surpasses all existing methods, even including GPT3.5 and ChatGPT, and achieves state-of-the-art performance.

Abstract: The task of zero-shot commonsense question answering evaluates models on their capacity to reason about general scenarios beyond
those presented in specific datasets. Existing approaches for tackling this task leverage external knowledge from CommonSense Knowledge Bases (CSKBs) by pre-training the model on synthetic QA pairs constructed from CSKBs. In these approaches, negative examples (distractors) are formulated by randomly sampling from CSKBs using fairly primitive keyword constraints. However, two bottlenecks limit these approaches: the inherent incompleteness of CSKBs limits the semantic coverage of synthetic QA pairs, and the lack of human annotations makes the sampled negative examples potentially uninformative and contradictory. 

To tackle these limitations above, we propose Conceptualization-Augmented Reasoner (CAR), a zero-shot commonsense question-answering framework that fully leverages the power of conceptualization. Specifically, CAR abstracts a commonsense knowledge triple to many higher-level instances, which increases the coverage of the CSKB and expands the ground-truth answer space, reducing the likelihood of selecting false negative distractors. Extensive experiments demonstrate that CAR more robustly generalizes to answering questions about zero-shot 
commonsense scenarios than existing methods, including large language models, such as GPT3.5 and ChatGPT. Our code, data, and model checkpoints are available at https://github.com/HKUST-KnowComp/CAR.

CAR: ConceptullizaioonAAugeeeeee Reasoner for Zero-Shot Commonsense Question Answering
Weiqi Wang'; Tianqing Fang",, Wenxuan Ding', Baixuan Xu', Xin Liu',
Yangqiu Song', Antoine Bosselut?
Department of Computer Science and Engincering, HKUST, Hong Kong SAR, China NLLP Lab, School of Computer and Communication Sciences, EPFL, Switzerland (wwangbw, tfangaa, yqsong)@cse. ust. hk, antoine. antoine. www.eeobsseeeeeeeeeee ch
Abstract
Distractor Knowledge Sampling
CSKB
(Usually highly Incomplete.)
The task of zero-shot commonsense question answering evaluates models on their capac- ity to reason about general scenarios beyond those presented in specific datasets. Existing approaches for tackling this task leverage ex- ternal knowledge from CommonSense Knowl- edge Bases (CSKBs) by pre-training the model on synthetic QA pairs constructed from CSKBs. In these approaches. negative examples (dis- tractors) are formulated by randomly sampling from CSKBs using fairly primitive keyword constraints. However, two bottlenecks limit these approaches: the inherent incompleteness of CSKBs limits the semantic coverage of syn- thetic QA pairs, and the lack of human anno- tations makes the sampled negative examples potentially uninformative and contradictory.
PersonX played football game, xWant, take rest
Levicalization
Personx invites PersonY to house, x,ant, have drink
What does Person' want after played a football gamm?
rest
Groons inst
PersonX likes sing, xWant, show to others
B: have drink
yoor degetto
C: show to others
X
Whong
An example of constructing synthetic QA pains from CSKB (Ma et al., 2021). The simple heuristic used in this process can result in false negative options.
Figure 1:
from their training sets (McCoy et al., 2019; Ma et al., 2019; Zhou al., 2021; Wang et al., 2021). I). This discrepancy arises because fine-tuned PLMs often rely on spurious, dataset-specific correlations to learn a task rather than learning to fully leverage implicit commonsense knowledge required for rea- soning (Branco et al., 2021). For reasoning systems to be effective, though, they must be robust across domains and generalize beyond the specificities of individual datasets.
To tackle these limitations above, we pro- pose Concepuullization-Auuuu Augmented Reasoner (CAR), rero-shot commonsense question- answering framework that fully leverages the power of conceptualization. Specifically, CAR abstracts commonsense knowledge triple to many higher-level instances, which increases the coverage of the CSKB and expands the ground truth answer space, reducing the like- lihood of selecting felse-ne distractors. Extensive experiments demonstrate that CAR more robustly generalizes to answering ques- tions about zero-shot commonsense scenar- ios than existing methods, including large language models, such as GPT3.5 and Chat- GPT. Our code, data, and model checkpoints are available at http://gihub..com//IKUSST KnowComp/CAR.
To confront the generalization issue in com- monsense reasoning tasks, the task of zero-shot commonsense Question-Answering (QA) requires models answer questions for evaluation bench- marks without access to their corresponding train- ing data (Shwartz et al., 2020; Li et al., 2020). Among several methods that tackle this task, the most performant ones inject commonsense knowl- edge from CSKBs (Hwang et al., 2021: Jiang et al., 2021) into PLMs by fine-tuning them on synthetic QA pairs transformed from commonsense knowl- edge triples, where the head and relation are trans- formed to question, and the tail serves ground answer. Negative examples are randomly sampled with keyword-overlap constraints (Ma et al., 2021). Such knowledge injection benefits not only QA
1 Introduction
Pre-trained Language Models (PLMs; Devlin et al., 2019; Clark et al., 2020) fine-tuned on task-specific training sets achieve remarkable near-human per- formance on held-out test sets, yet struggle to gener- alize examples that are distributionally different
Equal Contribution

---

Original Knowledge Triple
memorizing specific knowledge. Conceptualiza- tion (He al., 2022) offers similar capability by abstracting a set of instances into concepts, which allows for the derivation of abstract commonsense knowledge associated with each concept that can be instantiated to assist reasoning on specific down- stream situations. For example, in Figure 2, "play football game" can be conceptualized as a tiring event, which further generalizes as abstract knowl- edge. The benefits of conceptualization are twofold. First, conceptualized commonsense knowledge in- troduces abstract knowledge through a one-step concept inference based on the original CSKB, enhancing knowledge coverage, Second, as the abstract knowledge is conditioned on the origi- nal knowledge, the recall of knowledge regarding the same head is increased, leading to more fine- grained constraints for negative option sampling. Inspired by these advantages, we propose CAR (Concepuulization-Auuuuu
(PersonX played' a football game, xWant, take rest)
Conceptualization Relations
Conceptualization Inference
(played football game, IaA, Sport) (played football garne, IsA, Tiring event)
(played football game, IsA, Exencise)
(Nr,,) (h,Iaa hg)
Abstract Knowledge
(PersonX (do) Sport, x,ant, take a rest)
(PersonX (de) Tiving event, x,antt, take a rest)
(PersonX (do) Exercise, xWant, take rest)
An example of conceptualization inference. More abstracted knowledge, such as (Do sport, xWant, take rest), can be obtained through conceptualization.
Figure 2:
tasks that are derived from CSKBs, such as So- cialIQA (Sap et al., 2019b), which is derived from ATOMIC (Sap al., 2019a), but also QA datasets in other domains (Bisk al., 2020).
Reasoner), a sim- ple yet effective zero-shot commonsense QA frame- work that leverages conceptualization to expand ex- isting CSKBs and reduce false-negative distractors. We first augment the original CSKB with concep- tualization infuse abstract commonsense knowl- edge to improve knowledge coverage. Then, we propose a conceptualization-cossriai sampling strategy that generates distractors with concept- level constraints to prevent false negative options (Section 4). Experimental results on five popular commonsense QA benchmarks demonstrate the ef- fectiveness of CAR, which even surpasses GPT3.5 and ChatGPT (Section 5). In Section 6, we ana- lyze why CAR works by providing human eval- uations that show a significant reduction of false negative options compared to other methods. Fi- nally, our analysis reveals that conceptualization- augmented training examples tend to be more am- bignous (S (Swayamdipta et al., 2020) than those pro- duced by prior heuristics, leading better out-of- domain generalization.
Despite recent advancements in this area, two major challenges remain. First, manually curated CSKBs, such as ATOMIC, are incomplete (Kuo and Hsu, 2010). While consolidating multiple CSKBs can improve coverage, it remains infea- sible to cover all conceivable knowledge for the vast range of entities and situations in the real world (He et al., 2022). Automatic methods for ex- panding CSKBs exist, such as knowledge base com- pletion (Li et al., 2016; Malaviya al., 2020), and knowledge distillation from large language mod- els (West et al., 2022: Gao et al., 2023), but they ei- ther fail to provide knowledge about novel entities or only provide highly accurate yet less informative knowledge (e.g., vague adjectives, such as happy, as situation descriptors). Second, in zero-shot com- monsense QA, negative examples are required for models to learn to distinguish the validity of com- monsense scenarios (Chen et al., 2023a). However, existing negative QA examples are synthesized us- ing simple heuristic-based negative sampling with- out considering deeper semantics, resulting in too many false negative options. For instance, in Fig- ure 1, "have a drink" is also plausible in the context of "after playing football game." These questions that label plausible options negative instances confuse the model during training, impeding its ability discern correct commonsense knowledge.
Related Works
Zero-shot Commonsense QA. Zero-shot com- monsense QA evaluates a model's reasoning gener- alizability on unseen QA entries without any super- vision signals from the corresponding annotated training data. To tackle this task, two primary pipelines have emerged in existing works. The first paradigm employs off-the-shelf language mod- els without changing the parameters, either using
We tackle both of these challenges by utilizing conceptualization. As Murphy (2004) posits, hu- mans rely on conceptual induction to draw infer- ences about unseen situations without the need for

---

vanilla language modeling with prompts (Trinh and Le, 2018; Li et al., 2022), with some inference- time mechanisms specifically designed for reason- ing, such as self-talk (Shwartz et al., 2020), cloze translation (Dou and Peng, 2022), and dynamic generation of reasoning sub- gubbgraphs and graph rea- soning (Bosselut et al., 2021). The second pipeline leverages external CSKBs as knowledge sources provide PLMs with additional supervision sig- nals for further fine-tuning (Banerjee and Baral, 2020; Ma et al., 2021; Su al., 2022). A common strategy involves converting knowledge triples in CSKBs to synthetic QA pairs by transforming the head and relation to a question, the tail to a gold answer, and (randomly) sample tails from other heads as distractors. Such fine-tuning paradigm benefits from incorporating CSKBs within differ- ent domains (Kim al., 2022: Shi et al., 2023) and exploiting multi-hop graph structures with graph neural networks (Guan et al., 2023), and heightens the model's s commonsense sensitivity in a QA con- text, which leads to state-of-the-art performances.
expand the size and diversity of training set with- out requiring costly data annotations (Wei and Zou, 2019). Various methods have been proposed to augment textual data, including those using ran- dom perturbation (Wei and Zou, 2019), text em- beddings (Wang and Yang, 2015), lexical seman- tics (Niu and Bansal, 2018), back translation (Sen- nrich et al., 2016), and large language models (West et al., 2022; Ismayilzada and Bosselut, 2023; Gao et al., 2023) for CSKB construction. Nevertheless, text- next-perturbation-aaseeee augmentations do not pro- vide new knowledge to CSKBs, and knowledge mining from large language models suffers from high typicality (e.g.. favoring simple commonsense over informative yet rare commonsense) and low density, still making negative sampling subject to false negatives (Malaviya et al., 2020).
3 Problem Definition
3.1 Definitions
Conceptualization. Formally, denote CSKB as D with knowledge triples in the format of D = {(hrr, r,,r E H,r E R, t E T}, where H, R. and T are the sets of heads, relations, and tails in the original CSKB. Following He et al. (2022), the conceptualized CSKB, conditioned on D, can be denoted as DC = {(he, r, the E He, E R, t e T}. where He is the set of conceptualized head events. Specifically, each conceptualized head ha is obtained by replacing a component i h with its abstract concept e while ensuring that the formed (he, r, t) triple is still plausible in the original con- text (r, t). Such (he, T, t) triples are commonly referred to as abstract commonsense knowledge.
Conceptualization. Conceptualization refers to the process of ubstracting group of instances or events into a general concept (Song et al., 201 1, 2015). In commonsense reasoning, it simulates conceptual induction (Murphy, 2004) and enables the derivation of abstract commonsense knowledge under the specific contextualization of the origi- nal commonsense knowledge (Tenenbaum et al., 2011), which is often lacking in existing CSKBs. Around many existing works studying conceptu- alization (Durme et al., 2009; Gong et al., 2016; Liu et al., 2022: Peng et al., 2022), He et al. (2022) investigate it at event-level semanties and construct AbSMICC an event conceptual- ization benchmark and knowledge base based on ATOMIC (Sap et al., 2019a). Recently, Wang et al. (2023a) propose to conceptualize CSKBs at scale with semi-supervised learning and demonstrate ab- stract knowledge can enhance commonsense in- ference modeling (Bosselut et al., 2019; Da et al., 2021). With current works mostly investigating the problem of conceptualization itself, none of them have extrinsically evaluated the impact of conceptualization on downstream tasks, such as commonsense QA (Talmor et al., 2019) or machine reading comprehension (Nguyen et al., 2016).
Zero-shot Commonsense QA. In this paper, we employ the zero-shot commonsense QA task pro- posed by Ma et al. (2021) to study our frame- work. First, the CSKB D is transformed into multi- ple (Qi, A:) pairs where Q2 is a natural langauge question and A, = {Ac1, A1,,,,,,,,. ..., ...,...,,sa Aim) is a set of options with m candidates. Specifically, for a given knowledge triple (h, () C D, we con- vert h, r into Qi via natural language templates and use as the ground answer. Additionally, we retrieve m distractors from other triples sam- pled from D using a manually defined strategy, such as keyword overlap filtering. The objective of our task is to train QA model from the syn- thetic QA sets D = {(Q,, A4) ri, ti) E D}. Once trained, the model is tested on held-out test
Data Augmentation. Data augmentation aims at generating new examples from existing data to

---

Distractor Knowledge Sampling
enjoy running, xifant,
shout out loud, awant,
Persont casino, xilant, hove drink
at office, MRaat, meet his colleague
CSKB
keep fit
Conceptualized
Conceptijalizaton
Conceptiplization
Conceptiilization
Conceptialication
CSKB
enjoy (sport). xifant, keep fit
[shoutinal, xwant,
Personx is [entertainment place), xiiant, have drink
Personk [arrival], meet his colleague
Step 1 Conceptualization Augmentation
No Conflict! Select distructors!
keep
cry
Concept Overlop [entertainment
Concept Keyword Overíop arrive, [arrival]
PersonX arrive the bar, xWant, relax himself
Step 2 Concept-Constrained Distractor Sampling
QA Pair Synthesis (Positive mnowledee
Conceptualization
PersonX arrive at the bar, what does PersonX
Pre- Trainnin
want to do?
Zero- Zero-Shat
PersonX arrive at the (entertainment place), xWant,, relax himself
A: relox himself B: keep fit Ct
PersonX arrive at (entertainment place), what does PersonX want to do? relox himsel/ keep fit C: cry
QA Benchmarks
[arrival], z,ant, relax himself
An overview of the CAR framework, which shows the process of synthesizing (PersonX arrive the bar, xWant, relax himself) into QA pairs. The triple is conceptualized first, and potential distractor triples are sampled and filtered by keyword and concept overlap. Only those triples that have no overlap are used as distractors.
Figure 3:
entries (Qtest, Atest) from QA benchmarks. This requires the model to perform zero-shot common- sense reasoning since the training data from the target benchmarks are unavailable to the model.
CAR Framework
This section introduces our proposed CAR frame- work. A general sketch is presented in Figure 3. Our framework can be summarized into three steps: (1) Conduct one-step conceptualization inference on existing triples in the CSKB to obtain abstract commonsense knowledge triples. (2) Transfer the triples into QA pairs and generate distractors using keywords and conceptualizations constraints. (3) Train the QA model using marginal ranking loss.
3.2 Dataset
We use ATOMIC (Sap et al., 2019b) the source CSKB D. ATOMIC contains inferential common- sense knowledge, in the format of (h, t) triple, that is associated with commonly seen events. Specifically, the heads of ATOMIC triples are events, whereas the tail nodes are either events or attributes. For conceptualization, we use the annotatdd abstract knowledge from Abstrac- tATOMIC (He et al., 2022) to train generative con- ceptualizer for acquiring D More details of con- ceptualizations and statisties of ATSMractATOMIC are provided in Section 4.1 and Appendix B.1.
4.1 Conceptualization Augmentation
To incorporate abstract knowledge into the CSKB, we begin by augmenting the (h, r, E D triples by conducting a one-step conceptualization inference. Initially, given a head event h, we retrieve all plau- sible conceptualizations Ch = (c,,,1 C1,,1,C,,,2 2..2.... for all identified instances i E {L1, f2. E h} using entity-linking heuristies to retrieve concepts from Probase (Wu al., 2012) and WordNet (Miller, 1995). The conceptualized head event h2 is then obtained by replacing an E with one of its re- trieved conceptualization e E (ci,1, C........ This is done for all identified instances and their re- trieved conceptualizatioos, thereby constructing the set of conceptualized head events of h. Subse- quently, we link the non-abstract counterpart (r. t) after ht to generate candidate abstract knowledge triples (he r, r)t, where we adopt a discriminator trained with a semi-supervised conceptualization- instantiation framework to determine their plausi- bility (Wang et al., 2023a). Only plausible triples
3.3 Evaluation Benchmarks
Following Ma al. (2021), we evaluate our frame- work on the validation split of five commonsense QA benchmarks: Abductive NLI (aNLI; Bhagavat- ula et al., 2020), CommonsenseQA (CSQA; Tal- mor et al., 2019), PhysicalIQA (PIQA; Bisk et al., 2020), SocialIQA (SIQA; Sap et al., 2019b), and WinoGrande (WG; Sakaguchi et al., 2021). These manually constructed benchmarks evaluate various knowledge types essential for robust commonsense reasoning (Kim et al., 2022). Detailed statistics and explanations of these benchmarks are provided in Appendix A.

---

are kept to form DC, Details about the conceptu- alization retrieval processes and the discriminator are presented in Appendix B.1.
"PersonX arrives at the bar, as a result, PersonX want to, relax himself." We then repeatedly mask out token at one time and calculate the masked loss. The final MLM score for an input sequence Te T2, with n tokens is:
4.2 Conseet-Constrainned QA Synthesis
To synthesize commonsense triple (h, r, 1) into a (Qi; A;) pair, we first transfer h, r into Q; by using natural language templates and set t as the ground- truth answer A,. For example, the triple in Figure 3 becomes "PersonX arrives at the bar, what does Per- sonX want to do?" with the answer being "relax himself." Additional distractors are generated by transforming sampled distractor triples from the original CSKB, where only triples with the same commonsense relation r are sampled to ensure in- formativeness. To prevent sampling false negative options, we constrain sampling distractor knowl- edge by filtering keywords and conceptualizations, Formally, denote the keywords of head event h Th 12,, and the full set of plausi- ble conceptualizations for all identified instances h as C₁ (ct,1, C,,,,, ... C.2.1s'... we as- sociate a triple r,,r,)) with Tn + C₁ to form its constraint. Only knowledge triple (h', r, t') which satisfies (Th- + Ci) n (Th + CA) = @ can be sampled as distractor candidate. This constraint requires that the two triples have no common key- words, and their instances cannot be abstracted into the same conceptualization. For example, in Figure 3, "(PersonX is the casino, xWant, have a drink))" cannot be used as a distractor triple because "casino" can be conceptualized as "entertainment place," which is the same as "bar" in the original triple. Finally, we sample two distractor triples for the triple (h, r) and use the tails of these two triples the distractors. To guarantee that the ab- stract commonsense knowledge from our previous augmentation is learnable by the QA model, we synthesize both the original triple (h, r, f) and its conceptualized versions (her r, t) into QA pairs.
1 S(T) 72 Σ
log P( 1. 1.111....) (1)
After calculating the scores S1, S2,.. for all answer candidates A1, A2,... marginal ranking loss based on Equation 2, where represents the margin and y is the index of the correct answer.
compute the
L=
Σ max Sy S.) m i=Liy
(2)
During the evaluation phase, we use the same scoring procedure to assign a score to each option and select the one whose concatenated sentence achieves the lowest score as the model's prediction.
5 Experiments
5.1 Setup
Baselines First, we use random voting (Ran- dom) and most-frequent labeling (Majority) to demonstrate the characteristics of each benchmark. Vanilla RoBERTa-Large (Liu et al., 2019), and DeBERTa-v3-Large (He et al., 2023) PLMs are used to demonstrate the power of fine-tuning. The performances of these two models under super- vised training regime are also included to show the upper bound of our results. We also include the results of several existing approaches that tackle the same task, including Self-talk (Shwartz et al.. 2020), COMET-DynaGen (Bosselut et al., 2021), SMLM (Banerjee and Baral, 2020). MICO (Su et al., 2022), and the previous state-of-the-ar STL- Adapter (Kim al., 2022). Most importantly, we compare our framework with Ma et al. (2021) to validate the efficacy of conceptualization since both methods share similar model architecture and training procedures. Both RoBERTa-Larr and 1-BERTa-v3-Large are used as the backbones for fair comparisons. There are, in total, 534,833 syn- thetic QA pairs provided by Ma et al. (2021).
4.3 Model Training
Following Ma et al. (2021), we train our QA model by fine-tuning a pre-trained Masked Language Model (MLM) using the Marginal Ranking (MR) loss. Let C represent the original context (if any), Q represent the question, and (A1, Ag...) be the list of options. We first concatenate C, Q, and an answer option A together via natural language prompts to generate input sequences (7,, T2, ...). For example, the synthesized question with its cor- rect answer in Figure 3 will be transformed as:
With the recent advances in Large Langauge Models (LLMs) (Bang al., 2023; Chan et al., 2023; Qin et al., 2023), we also benchmark the performances of GPT3.5 (Brown et al., 2020) and ChatGPT (OpenAI, 2022) as baselines. We prompt

---

Model
CSKB
a-NLI
CSQA
PIQA
SIQA
WG
Avg.
Random
50.0
20.0
50.0
33.3
50.0
40.7
Majority
50.8
20.9
50.5
33.6
50.4
41.2
RoBERTa-L (Liu et al., 2019)
65.5
45.0
67.6
47.3
57.5
56,6
DeBEKTa-v3-L (Ile al., 2023)
59.9
25.4
44.8
47.8
50.3
45.6
Self-talk (Shwartz et al., 2020)
32.4
70.2
46.2
54.7
COMET-DynGen (Bosselut al. 2021)
ATOMIC
50.1
SMLM (Banerjec and Baral, 2020)
65.3
38.8
48.5
MICO (Su et al., 2022)
ATOMIC
44.2
56.0
STI.-Adapter (Kim al., 2022)
ATOMIC
71.3
66.5
71.1
64.4
60.3
66.7
Backbone: RoBERTa-Large 340M
RoBERTA-L (MR) (Ma al., 2021)
ATM-10X
70.8
59.4
72.1
58.5
58.3
63.8
A RoBERTa-L (MR) (Ma et al., 2021)
ATOMIC
70.8
64.2
72.1
63.1
59.2
65.9
CAR-RoBERTa-LL (Ours)
ATOMIC
72.3
64.840.6
73.2
64.8F1.7
61.3
67.3
CAR-RoBERTa-L (Ours)
ATM
72.7+1.9
66.3
73.2
64.0qu.9
62.0F2.8
67.6p1.7
Backhone: DeBERTaa-v3-Laagg 435M
DeBERTa-v3-L (MR) (Ma et al., 2021)
ATM-10X
75.1
71.6
79.0
59.7
71.7
71.4
A DeBERTa-v3-1 (MR) (Ma al., 2021)
ATOMIC
76.0
67.0
78.0
62.1
76.0
71.8
CAR-DeBERTn-v3-L. (Ours)
ATOMIC
78.9+2.9
67.2
78.6
63.8
78.
73.3
CAR-DeBERTa-2--L (Ours)
ATM
79.6f3.6
69.3
78.6fo.6
64.0,1.9
y8.2
73.9
Large Language Models
GPT-3.5 (exc--davinc1-0033
61.8
68.9
67.8
68.0
60.7
65.4
ChatGPT 1.. tutbbo)
69.3
74.5
75.1
69.5
62.8
70.2
Supervised Learning & Human Performance
RoBERTa-L (Supervised)
85.6
78.5
79.2
76.6
79.3
79.8
DeBERTa-v3-L (Supervised)
89.0
82.1
84.5
80.1
84.1
84.0
Human Performance
91.4
88.9
94.9
86.9
94.1
91.2
Table Zero-shot evaluation results (%) five commonsense question answering benchmarks. The best results are bold-faced, and the second-best ones are underlined. t indicates the performance gain of our framework (marked with o) compared with results acquired by Ma et al. (2021) on ATOMIC (marked with A). ATM stands for the ATOMIC with abstract commonsense knowledge injected. ATM-10X stands for using ATOMIC-10X (West et al.. 2022) as the source CSKB D). All baseline results are consistent with their original papers.
the LLM directly in zero-shot setting, where no in-context learning (Min et al., 2022) or chain-of- thought reasoning (Wei al., 2022) are applied. For every QA entry, the LLM is presented with question, several choices, and natural language command that asks to choose the index of the correct answer directly (Robinson et al., 2022). We then parse the generated outputs to obtain the "pre- dictions" of LLM by using meticulously designed rules and compare them with the ground-truth la- bels. More details of the baselines and LLM setups be found in Appendix B.2 and B.3.
QA pairs for training. We employ an AdamW opti- mizer (Loshchilov and Hutter, 2019) with a learn- ing rate of 7 and max sequence length of 128 to accommodate QA pairs with different lengths. We select the best checkpoint according the highest accuracy achieved on the synthetic validation QA set. Each experiment is repeated using three differ- ent random seeds, and the average performance is reported. The model is warmed up with 5% of total iterations and evaluated every 1000 global steps, while the margin for the marginal ranking loss is to 1, in line with the choices made by Ma et al. (2021) and Kim al. (2022). More details about implementations can be found in Appendix B.4,
Implementation Details We use accuracy as the evaluation metric and compare our framework with the following baseline methods. For con- ceptualization, we leverage an off-the-shelf con- ceptualizer from Wang et al. (2023a), which is semi-supervised conceptualization discrimina- tor fine-tuned labeled conceptualization data from AbstractATOMIC and unlabeled data from ATOMIC. We use a plausibility score T 0.9 to filter out plausible conceptualizations, which results in 440K conceptualization aided synthetic
5.2 Results
The main results are reported in Table 1. For the baselines, DeBERTa-v3-Large (MR) trained on ATOMIC achieves the best performance, fol- lowed by ChaIGPT. Both achieve an accuracy of more than 70% on average. Our best system, based on DeBERTa-v3- Large and trained on our conceptuliiaaiionaaugeeen ATOMIC, achieves

---

Augmentation
Divt
Exp.Divt
Plau.t
Ne
aNLI
CSQA
PIQA
SIQA
WG
N/A (Baseline)
N/A
N/A
88.0
45.7
76.0
67.0
78.0
62.1
76.0
EDA (Wei and Zou, 2019)
8,10
4.67
9.33
33.0
76.5
65.6
76.6
61.4
74.9
Word2Vec (Wang and Yang, 2015)
11.8
4.00
9.00
55.0
74.3
65.8
75.1
62.9
74.7
GLOVE (Wang and Yang. 2015)
8.21
6.67
4.67
44.3
74.7
64.2
74.6
61.1
74.4
BERT-base (Kohayashi, 2018)
0.81
8.33
14.3
41.7
70.4
63.9
72.4
633.5
61.0
Synonym (Niu and Bansal, 2018)
6.92
11.0
5.67
45.0
75.5
64.9
74.5
62.5
75.7
GPT3-distil (West et al., 2022)
35.6
24.3
95.7
42.7
75.4
71.8
75.6
63.4
76.0
Conceptualization (Ours)
48.5
37.0
90.0
22.7
79.6
69.3
78.6
64.0
78.2
Table 2: Comparison results (%) of different augmentation methods against conceptualization. N/A stands for not ratio of plausible augmented knowledge, %F.Neg. represents
using augmentation. Plau. is the the expert-annotated proportion of false negative options. Div. and Exp.Div. are diversities measured by embedding similarity and expert annotated knowledge coverage. Performances the right refer to accuracies achieved by the QA model trained on data augmented by each method. The best performances are bold-faced.
state-of-the-art results and significantly outper- forms all PLM-based baselines every bench- mark, and can advance the average accuracy by 2.1% compared with the same baseline model. also significantly surpasses the performance of the same model that is trained on ATOMIC-10X with only 10% amount of data (more explanations and experiments in Appendix B.5). Notably, compared with LLMs, our system champions three bench- marks and performs better on average with 3.7% leap. This indicates that supervision signals from CSKBs are important for downstream applications, and CSKBs aided by conceptualization can signifi- cantly enhance this process. Moreover, as an abla- tion, we study the role of concept-level distractor sampling by discarding conceptualization augmen- tation and only training the models on ATOMIC, synthesized to QA format with our proposed con- straint technique. Comparing the results in Table 1, it can be observed that the concept-level distrac- for sampling improves the average performance by approximately 1.5%. This demonstrates that our proposed technique effective, and generating dis- tractors with a stronger positive knowledge recall is helpful in synthesizing QA pairs that are both fair and informative.
ful for training. Finally, we study the impact of filtering ATOMIC10X with different critic thresh- olds, the ablations of CAR, and the effect of concep- tualization from an out-of-domain generalization perspective in Appendix B.5, B.7, and B.8.
6.1 Comparisons With Data Augmentations
To demonstrate the effectiveness of our proposed conceptualization method, we conduct comprehen- sive analyses with other data augmentations that expand the semantic coverage of CSKBs in simi- lar way conceptualization using both expert and automatic evaluations. We use EDA, augmenting with word embedding (Word2Vec; Mikolov et al., 2013 and GLOVE; Pennington et al., 2014), con- textual embedding (BERT; Devlin et al., 2019), and synonym (WordNet; Miller, 1995) as baselines. To align all the baselines for fair comparisons, we only augment the identified instance i E h in each ATOMIC triple's head event h according to the number of its valid conceptualizations |Ca]. Addi- tionally, we randomly sample another same amount of knowledge from ATOMIC- 10X into ATOMIC a form of augmentation by distilling GPT3 (Brown et al., 2020) to set distilling an LLM as another baseline (more explanations in Appendix B.5).
6 Analysis and Discussion
We comprehensively analyze the comparison us- ing three dimensions: diversity, quality of synthetic QA pairs, and zero-shot commonsense QA perfor- mances. Three expert annotators are recruited to facilitate our evaluations who are undergraduate or graduate students actively involved in common- sense research. They demonstrate high level of agreement among themselves, with an IAA of 83% in terms of pairwise agreement and Fleiss Kappa score (McHugh, 2012) of 0.64, comparable to 0.62, as reported by (Ma et al., 2021).
In this section, we study the effects of conceptual- ization and the reasons contributing to CAR's 's suc- cess. First, we conduct expert evaluations on the synthetic QA pairs to study the quality and diversity of different CSKB augmentation methods in com- parison with conceptualization. Second, we con- duct training dynamics (Swayamdipta et al., 2020) analysis to show that conceptualization aided QA pairs can provide more ambiguous examples help-

---

ATraining Dynamics of ATOMIC Knowledge
Training Dynamics of
Abstract Knowledge vs. ATOMIC- 10X Knowledge
#data (log)
10"
10"
10
10
Conceptusisatiio
AT0MIC-10X
10
10
10
10
10"
10
10
10
0.4
0.2 0.0
0.2
0.8 0.0
0.4
0.8
0.0
0.1
0.2 Variability
0.3
0.25
0.50
0.75
1.00
Aarriability
AConfidence
Confidence
Figure 4: Analyses on training dynamics different knowledge. The dotted lines refer to the median values.
Diversity. First, we study whether augmentations introduce new knowledge the training set. We begin by calculating the average cosine simi- larity of each ATOMIC triple and its augmented siblings from each method according to their Sen- lenceBERT (Reimers and Gurevych, 2019) em- beddings. For ATOMIC-10X, we regard the sam- pled knowledge as augmentations. The comple- ment of average similarity across all ATOMIC triples serves as automatically measured diver- sity (Div.). Meanwhile, we retrieve top- 10 similar triples from ATOMIC for each augmented triple ac- cording to their SentenceBERT similarity. The ex- perts are asked to annotate whether each triple can be semantically covered by their retrievals. We de- fine the expert-evaluated diversity as the ratio of un- covered triples among 300 samples. Table 2 shows that conceptualization champions both metrics, in- dicating that the introduced abstract knowledge is diverse and lacking in existing CSKBs, which is helpful expanding their knowledge coverage.
accuracy reported by Ma et al. (2021).
Zero-shot Commonsense QA Performances. Finally, we train DeBERTa-v3-Large models on the QA pairs synthesized from the concatenation of both original and augmented ATOMIC triples from each method. Only keywords of each head event used as their constraints. The models trained using marginal ranking loss, as explained in Sec- tion 4.3, and evaluated on five QA benchmarks in a zero-shot manner. Performances by different methods are shown in Table 2. We observe that conceptualization outperforms all other augmen- tations on average and successfully improves the model's zero-shot commonsense reasoning ability.
Comparison with ATOMIC-10X. Augmenting ATOMICI0X appears to be a promising option as it contains a wealth of valuable commonsense knowledge. However, despite its diverse and high- quality knowledge, Table 2 demonstrates that the model cannot leverage this information effectively. One possible explanation is that the model's perfor- mance is hindered by the significantly high number of false-negative distractors. This issue arises be- cause the knowledge distilled from GPT-3 tends to be versatile, resulting in many tail events being general and vague. These events can be applied to large collection of heads, which leads to false neg- ative options. More experiments and case studies are in Appendix B.5 and C, respectively.
Quality of Synthetic QA Pairs. Next, we syn- thesize the augmented triples into QA pairs with their head events' keywords and augmentations as constraints. We then sample 300 QA pairs for each method and ask the same experts to perform expert evaluations by annotating the correctness of each QA pair's ground-truth answer and whether the dis- tractors could also be plausible with respect to the augmented head event. This evaluates the plausi- bility ratio of the augmented knowledge and the ratio of QA pairs containing false negative distrac- tors. Table 2 shows that the majority of augmented knowledge is implausible, and they fail to enhance the
6.2 Training Dynamics Analysis
Training dynamics effectively assess a model's 's confidence and variability for individual instances when training a large dataset. In the context of QA, we define confidence as the model's certainty when assigning the correct label to the ground-truth option compared to distractors, as indicated by the logit difference. Variability, on the other hand, refers to the fluctuation of confidence over time. These insights can aid in analyzing the model's
distractors sampling. Conceptualization, other hand, maintains being highly plausible and can effectively eliminate false negative distractors. Expert annotators also achieve remarkable ac- curacy of 86% while working on 300 randomly sampled question- question-answer answer pairs, surpassing the 80%

---

Model
CSKB
Avg.
74.00
Accuracy
73.75
RoBERTa-L (MR) (Ma et al., 2021)
CWWV
64.8
MTL (Kim et al., 2022)
CWWV
63.7
73.50
ZS-Fusion (Kim al., 2022)
CWWV
64.7
73.25
BAR-RoBERTa-LE (Ours)
CWwV
65.8
73.00
50
60
70
80
90
100
Table 3: Experiments on the generalizability of CAR on other CSKBs (CWWV).
Proportion of ATM data (%)
Figure 5: Average accuracy achieved by models trained on our training set downsampled to several ratios.
behavior when different knowledge is introduced into the training set. More explanations are Ap- pendix B.6.
6.3 Impact of Training Data Size
In Figure 5, we present the influence of the number of training examples against the final performance, which reveals clear and intuitive trend of a pos- itive correlation between the amount of training data and overall performance.
In this section, we examine the impact of ab- stract commonsense knowledge (conceptualiza- tion) and GPT3-distilled knowledge (ATOMIC- 10X) by exploring their training dynamics on two sets of data. We train three QA models on syn- thetic QA pairs from conceptualization -uggeente ATOMIC, -rogietoX-augmmentee ATOMIC, and the original ATOMIC, which serves as the base- line. First, we randomly select the same 1,000 QA pairs synthesized from the original ATOMIC and calculate their training dynamics using these three models. The left side of Figure 4 displays the alterations caused by the two augmentation meth- evident
6.4 Generalization to other CSKBs
We explore the feasibility of transferring our frame- work to CSKBs other than ATOMIC. We take the CWWV dataset as an example, which comprises multiple CSKBs, including ConceptNet (Speer et al., 2017), WordNet (Miller, 1995), and Wiki- Data (Vrandecic and Krötzsch, 2014). We use the off fff-the-shelf GPT2 conceptualizer (Wang al., 2023a) and ChatGPT as two flexible generative conceptualizers. The generated conceptualizations are then transformed into abstract knowledge and integrated into the CWWV dataset. The experimen- tal results are presented in Table 3, which shows an improvement of over 1% compared to all baselines leveraging CWWV as the source of knowledge, in- dicating CAR's generalizability to other CSKBs. More details are presented in the Appendix B.9.
ods in comparison with the baseline. that introducing abstract commonsense knowledge through conceptualization significantly reduces the model's s average variability and enhances its confi- dence in learning the knowledge from the original ATOMIC. In contrast, incorporating knowledge from ATOMIC-10X produces the opposite effect.
Second, we check the training dynamics on 1,000 randomly sampled QA pairs synthesized from abstract commonsense knowledge and an- other 1,000 from knowledge in ATOMIC-10X. The rightmost plots in Figure 4 reveal that, com- pared to ATOMIC-10X, conceptualization intro- duces knowledge with higher variability and lower confidence, making more umbiguous and chal- lenging for the model to learn. As Swayamdipta et al. (2020) suggest, such data contributes to a more robust model to out-of-distribution (OOD) data, which are downstream QA datasets in our conclude that conceptualiza- case. Therefore, tion is superior ATOMIC-10X abstract knowl- edge, on the one hand, makes the original knowl- edge more easy-to-learn to aid optimization, and on the other hand, provides more ambiguous examples to boost OOD generalization.
7 Conclusions
In this paper, we present CAR, pioneering frame- work for zero-shot commonsense QA empowered by conceptualization. Our approach surpasses even large language models on five QA benchmarks, achieving state-of-the-art performance on average. Our analyses reveal that conceptualization can im- prove the sampling of negative examples, and ab- stract knowledge is more helpful compared with those distilled from GPT3 it provides more am- biguous knowledge to support OOD generalization. These findings demonstrate the substantial bene- fits of introducing conceptualization and abstract knowledge into zero-shot commonsense reasoning.

---

Limitations
well-trained with specially designed instructions and have voluntarily agreed participate without compensation. Based on this, the authors believe that this paper does not raise any ethical concerns to the best their knowledge.
One limitation of this paper is that the proposed CAR framework has only been validated on the ATOMIC datasct. While previous works (Ma et al., 2021; Kim et al., 2022: Dou and Peng, 2022) have studied the zero-shot commonsense question swering task by consolidating multiple CSKBs, including ATOMIC (Sap et al., 2019a), Concept- Net (Speer et al., 2017), WordNet (Miller, 1995), VisualGenome (Krishna et al., 2017), and Wiki- Data (Vrandecic and Krötzsch, 2014), our work only utilizes ATOMIC (more details discussed in Appendix B.2). This was mainly due to the avail- ability of conseptualizations for the CSKB, with only AbstractATOMIC (He et al., 2022) being avail- able as the conceptualized expansion of ATOMIC, while other CSKBs lack such resources. Addition- ally, ATOMIC has been shown to play the most critical role in experimental results by Ma et al. (2021). Nonetheless, such limitation does not re- strict CAR's potential to seek further improvements from incorporating other CSKBs, as conceptualiza- tion frameworks, such as CAT (Wang al., 2023a), be applied other CSKBs and provide the re- quired resources for CAR to operate. Thus, we be- lieve CAR can overcome such limitations and still possess the potential to improve with more CSKB- associated conceptualization resources available.
Acknowledgements
The authors would like to thank Haochen Shi for his help in implementing the training dynam- ics and the anonymous reviewers for their con- structive comments. The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20), and the GRF (1621 1520 and 16205322) from RGC of Hong Kong. We also thank the support from the UGC Research Match- ing Grants (RMGS20EG01-D,, RMGS20CRI1 1, RMGS20CRI2, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08), We also grate- fully acknowledge the support of the Swiss Na- tional Science Foundation (No. 215390), Innosu- isse (PFFS-21-29), the EPFL Science Seed Fund, the EPFL Center for Imaging, Sony Group Corpo- ration, and the Allen Institute for Al.
References
Jiaxin Bai, Xin Liu, Weiqi Wang, Chen Luo, and Yangqiu Song. 2023. Complex query answering on eventuality knowledge graph with implicit logical constraints. CoRR, abs/2305.19064
Ethics Statement
This paper presents CAR, a novel framework for zero-shot commonsense question answering that achieves state-of-ut of-thc--ttt-tttt performance via concep- malizatioon. All datasets used, including ATOMIC, AbstracIATOMMC,, and commonsense question- answering benchmarks, are publicly available and shared via open-access licenses solely for research purposes, consistent with their intended usage. These datasets are anonymized and desensitized, ensuring that no data privacy issues are involved. Moreover, the CAR framework is a question- answering system that selects the most plausible choice from a list of options and does not yield any private, offensive, biased, or sensitive infor- mation or social and political issues. The expert unnotations are performed by the authors of this pa- per part of their contribution, who are graduate and undergraduate students working on machine commonsense in natural language processing, and they are fully aware of the annotation protocol and the intended use of their annotations. They are
Pratyay Banerjee and Chitta Baral. 2020. Self- supervised knowledge triplet learning for zero-shot question answering. In Proceedings the 2020 Con- ference on Empirical Methods Natural Language Processing, EMNLP 2020, Online. November 16-20, 2020. Association for Computational Linguistics.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie. Holy Lovenia. Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do. Yan Xu, and Pascale Fung. 2023. multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hal- lucination, and interactivity. CoRR, abs/2302.04023.
Chandra Bhagavatula. Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In The

---

Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Secone Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, Febrwary 7-12, 2020. AAAT Press.
Zeming Chen, Gail Weiss, Eric Mitchell, Asli Celiky- ilmaz, and Antoine Bosselut. 2023b. RECKON- ING: reasoning through dynamic knowledge encod- ing. CoRR, abs/2305.0633
Kevin Clark, Minh- Thang Luong, Quoe V. Le, and Christopher D. Manning. 2020. ELECTRA: pre- training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 36-30, 2020. OpenReview.net.
Antoine Bosselut, Ronan Le Bras, and Yejin Choi. 2021. Dynamic neuro-symin -ymuroli knowledge graph construc- tion for zero-shot commonsense question answering. In Thirty-Fifth AAAI Conference on Artificial Intel- ligence, AAAI 2021. Phirty-Third Conference on In- novative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2021, Virtual Event February 2-9, 2021. AAAT Press.
Jeff Da, Ronan Le Bras, Ximing Lu, Yejin Choi, and Antoine Bosselut. 2021. Analyzing commonsense emergence in few-shot knowledge models. In 3rd Conference Automated Knowledge Base Construc- tion, AKBC 2021, Viruual, October 4-8, 2021.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Conference of the Association for Com- putational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. Association for Computational Linguistics.
Zheye Deng. Weiqi Wang. Zhaowei Wang. Xin Liu, and Yangqiu Song. 2023. Gold: A global and local-aware denoising framework for commonsense knowledge graph noise detection. CoRR, abs/2310.12011
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and
Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers). Association for Computational Linguistics.
Ruben Branco, António Branco, João António Ro- drigues, and João Ricardo Silva. 2021. Shortcutted commonsense: Data spuriousness in deep learning of commonsense reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Puntta Cana, Dominican Republic, 7-11 November. 2021. Association for Computational Linguisties.
Wenxuan Ding, Shangbin Feng, Yuhan Liu, Zhaoxuan Tan, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2023. Knowledge crosswords: Geomet- ric reasoning over structured knowledge with large language models. CoRR, abs/23 10.01290.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alee Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020. December 6-12, 2020, virtual.
Zi-Yi Dou and Nanyun Peng. 2022. Zero-shot com- monsense question answering with cloze translation and consistency optimization. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Fitirty-Fourte Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial In- relligence, FAAI 2022 Virtual Event, February 22 March 1, 2022. AAAI Press.
Benjamin Van Durme, Phillip Michalak, and Lenhart K. Schubert. 2009. Deriving generalized knowledge from corpora using wordnet abstraction. In EACL 2009. 12th Conference of the European Chapter of the Association for Computational Linguistics, Pro- ceedings of the Conference, Athens, Greece, March 30 April 3, 2009. The Association for Computer Linguistics.
Chunkit Chan, Jiayang Cheng, Weiqi Wang, Yuxin Jiang, Tianqing Fang. Xin Liu, and Yangqiu Song. 2023. Chatgpt evaluation on sentence level relations: A focus on temporal, causal, and discourse relations. CoRR, (et/2304.14827
Jiangjie Chen, Wei Shi. Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. 2023a. Say what you mean! large language models speak too positively about negative commonsense knowledge. In Proceedings of the st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Pupers), Toronto, Canada, Association for Computational Linguistics.
Tianqing Fang, Weiqi Wang, Schyun Choi, Shibo Hao, Hongming Zhang, Yangqiu Song, and Bin He. 2021a. Benchmarking commonsense knowledge base pop- ulation with an effective evaluation dataset. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2021,

---

Virtual Event Punta Cana, Dominican Republic, 7- 11 November, 2021. Association for Computational Linguistics.
Filip llievski, Pedro A. Szekely, and Bin Zhang. 2021. CSKG: the commonsense knowledge graph. In The Semantic Web- 18th International Conference, ESWC 2021, Virtual Event, June 6-10, 2021, Proceed- ings, volume 12731 of Lecture Notes in Computer Science. Springer.
Tianqing Fang, Hongming Zhang. Weiqi Wang. Yangqiu Song, and Bin He. 2021b. DISCOS: bridg- ing the gap between discourse knowledge and com- monsense knowledge. In WWW 21: The Web Con- ference 2021, Virtual Event/ / Ljubljana, Slovenia, April 19-2 23, 2021. ACM/ / IW3C2.
Mete Ismayilzada and Antoine Bosselut. 2023. kog- ito: A commonsense knowledge inference toolkit. In Proceedings the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics. EACL 2023 Systtm Demonstrations, Dubrovnik, Croatia, May 2-4, 2023, Association for Computational Linguistics.
Silin Gao. Beatriz Borges, Soyoung Oh, Deniz Bayazit, Saya Kanno, Hiromi Wakaki, Yuki Mitsufuji, and An- toine Bosselut. 2023. Peacok: Persona commonsense knowledge for consistent and engaging narratives. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), ACL Toronto, Canada, July 9-14, 2023. Association for Computational Linguisties.
Liwei Jiang, Antoine Bosselut, Chandra Bhagavatula, and Yejin Choi. 2021. "i'm not mad": Common- sense implications of negation and contradiction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, Associ- ation for Computational Linguistics.
Yu Gong, Kaiqi Zhao, and Kenny Qili Zhu. 2016. Repre- senting verbs as argument concepts. In Proceedings of the Thirtieth AAAT Conference on Artificial Intelli- gence, February 12-17, 2016, Phoenix, Arizona, USA. AAAI Press.
Yu Jin Kim, Beong-woo Kwak, Youngwook Kim, Reinald Kim Amplayo, Seung-won Hwang, and Jiny- oung Yeo. 2022. Modularized transfer learning with multiple knowledge graphs for sero-shot common- sense reasoning. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, NAACL 2022, Seartle, WA, United States, July 10-15, 2022. Association for Com- putational Linguistics.
Xin Guan, Biwei Cao, Qingqing Gao. Zheng Yin,
Bo Liu, and Jiuxin Cao. 2023. Multi-hop com- monsense knowledge injection framework for zero- shot commonsense question answering. CORR, abs/2305.05936.
Mutian He, Tianqing Fang, Weiqi Wang, and Yangqiu Song. 2022. Acquiring and modelling abstract com- monsense knowledge via conceptualization. CoRR, abs/2206.0155
Sosuke Kobayashi. 2018 Contextual augmentation: Data augmentation by words with paradigmatic rela- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, NAACL-HLL New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers). Association for Computational Linguistics.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTav3: Improving deBERTa using ELECTRA- style pre- vre-trainin with grudient-disentane ding sharing. In The Eleventh International Confer- ence on Learning Representations.
embed-
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Vi- sual genome: Connecting language and vision ing crowdsourced dense image annotations. Int. J. Comput. Vis., 123(1).
Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing Huang. 2019. Glossbert: BERT for word sense dis- ambiguation with gloss knowledge. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing, EMNLP-JJCNLP 2019, Hong Kong, China, November 3-7, 2019. Association for Computational Linguistics.
Yen-Ling Kuo and Jane Yung-jen Hsu. 2010. Bridg- ing common sense knowledge bases with analogy by graph similarity. In Collaboratively-Bute Knowl- edge Sources and Artificial Intelligence, Pupers from the 2010 AAAI Workshop, Atlanta, Georgia, USA, July 11, 2010. volume WS-10-02 of AAAI Technical Report. AAAL
Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da. Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2021. (comet-) atomic 2020: On sym- bolic and neural commonsense knowledge graphs. In Thirty-Fifth AAAI Conference on Artificial Intel- ligence, AAAI 2021, Thirty- Conference on In- novative Applications of Artificial Intelligence, IAAT 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence EAAI 2021. Virtual Event, February 2-9, 2021. AAAI Press.
Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume I: Long Papers. The Association for Computer Linguistics.

---

Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoff- mann, Cyprien de Masson d' A'ttumee Phil Blunsom, and Aida Nematzadeh. 2022. A systematic investiga- tion of commonsense knowledge in large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. EMNLP 2022, Abu Dhabi, United Arah Emirates, De- cember 7-11, 2022. Association for Computational Linguistics.
the 57th Conference of the Association for Computa- tional Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers. Association for Computational Linguisties.
Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3).
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013, Efficient estimation of word representa- tions in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scotrsdale, Arizona, USA, May 2-4, 2013, Workshop Track Pro- ceedings.
Zhongli Li, Wenhui Wang. Li Dong, Furu Wei, and Ke Xu. 2020. Harvesting and refining question- answer pairs for unsupervised QA. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, ACI. 2020, Online, July 5-10, 2020. Association for Computational Linguistics.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11).
Jingping Liu, Tao Chen, Chao Wang, Jiaqing Liang, Li- han Chen, Yanghua Xiao, Yunwen Chen, and Ke Jin. 2022. Voesk: Verb-oriented commonsense knowl- edge mining with taxonomy-gant guidedd induction. Artif. Intell., 310.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannameh Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arub Emirates, December 7-11, 2022. Association for Computational Linguistics.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 16692.
Gregory Murphy. 2004. The big book of concepts. MIT press.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrat- ing neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Infor- mation Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016. volume 1773 of CEUR Workshop Proceedings. CEUR-WS, WS..rg.
Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Ny- berg, and Alessandro Oltramari. 2019. Towards gen- eralizable neuro-symholli systems for commonsense question answering. CoRR, abs/1910. 14087.
Kaixin Ma, Filip Ilieskii, Jonathan Francis, Yonatan Bisk, Eric Nyberg, and Alessandro Oltramari. 2021. Knowledge-drive data construction for zero-shot evaluation in commonsense question answering. In Thirty-Fifth AAAI Conference Artificial Intelli- gence, AAAI 2021, Thirty-Thit Third Conference on In- novative Applications of Artificial Intelligence, IAAT 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence EAAI 2021. Virtual Event, February 2-9, 2021. AAAI Press.
Tong Niu and Mohit Bansal. 2018. Adversarial over- sensitivity and over-stability strategies for dialogue models. In Proceedings of the 22nd Conference on Computational Natural Language Learning. CoNLL 2018, Brussels, Belgium, October 31 November I, 2018. Association for Computational Linguistics.
OpenAL. 2022. Chatgpt: Optimizing language models for dialogue. OpenAl.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishknnn Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NewrIPS.
Chaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, and Yejin Choi. 2020. Commonsense knowledge base completion with structural and se- mantic context. In The Fhirth-Fourth AAAI Con- ference on Artificial Intelligence, AAAI 2020, The Thirty-Second Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial In- telligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press.
Hao Peng. Xiaozhi Wang, Shengding Hu, Hailong Jin, Lei Hou, Juanzi Li, Zhiyuan Liu, and Qun Liu. 2022. COPEN: probing conceptual knowledge in pre- trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2022, Abu Dhabi, United
Tom McCoy, Ellic Pavlick, and Tal Linzen. 2019, Right for the wrong reasons: Diagnosing syntactic heuris- tics in natural language inference. In Proceedings of

---

Arab Emirates, December 7-11, 2022, Association for Computational Linguistics.
with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACI. 2016, August 7-12, 2016, Berlin, Germany, Volume I: Long Papers. The Association for Computer Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL. Association for Computational Linguis- tics.
Haochen Shi, Weiqi Wang. Tianqing Fang, Baixuan Xu. Wenxuan Ding, Xin Liu, and Yangqiu Song. 2023. Qadynamics: Training dynamics-driven synthetic qa diagnostic for zero-shot commonsense question an- swering. CoRR, abs/23 10. 1303.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? CoRR, abs/2302.06476.
Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020. Online, November 16-20, 2020. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 188).
Yangqiu Song, Haixum Wang, Zhongyuan Wang, Hong- song Li, and Weizhu Chen. 2011. Short text concep- tualization using a probabilistic knowledgebase. In IJCAI 2011, Proceedings of the 22nd International Joint Conference on Artificial Intelligence, Barcelona, Catalonia, Spain, July 16-22, 2011 IICAI/AAII
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Lunguage Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNE 2019, Hong Kong, China, November 3-7, 2019. Association for Computational Linguistics.
Yangqiu Song, Shusen Wang. and Haixun Wang. 2015. Open domain short text conceptualization: A genera- tive + descriptive modeling approach. In Proceedings of the Twenty-Fourd International Joint Conference Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015. AAAI Press.
Joshua Robinson, Christopher Michael Rytting, and David Wingate. 2022. Leveraging large language models for multiple choice question answering. CoRR, abs/2210. 12353.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty- AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco. California, USA. AAAI Press.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: an adver- sarial winograd schema challenge at scale. Commun. ACM, 64(9).
Ying Su, Zihao Wang, Tianqing Fang, Hongming Zhang, Yangqiu Song. and Tong Zhang. 2022. MICO: A multi-alternative contrastive learning framework for commonsense knowledge representation. In Find- ings of the Association for Computational Linguistics: EMNLP 2012,, Abu Dhabi, United Arab Emirates, De- cember 7-11, 2022. Association for Computational Linguistics.
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan- dra Bhagavatula. Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith. and Yejin Choi. 2019a. ATOMIC: an atlas of machine commonsense for if-then reasoning. In The Thirty-Third AAAI Con- ference on Artificial Intelligence, AAAI 2019, The Thirty-Firs Innovative Applications of Artificial In- telligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2019, Honolulu, Hawaii, USA, January 27 February 2019. AAAI Press.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. Association for Computational Linguistics.
Maarten Sap, Hannah Rashkin Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social iqa: Common- sense reasoning about social interactions. In Proceed- ings of the 2019 Conference on Empiricul Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing, EMNLP-IJCN 2019, Hong Kong, China, November 3-7, 2019. Association for Computational Linguistics.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowl- edge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN, USA,
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models

---

June 2-7, 2019, Volume 1 (Long and Short Papers). Association for Computational Linguistics.
Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. NAACL 2022, Seattle, WA, United States, July 10-15, 2022. Association for Computational Linguistics.
Joshua B Temenbaum, Charles Kemp, Thomas L Grif- fiths, and Noah D Goodman. 2011. How to grow a mind: Statistics, structure, and abstraction. science, 331(6022).
Trieu H. Trinh and Quoe V. Le. 2018. A simple method for commonsense reasoning. CoRR, abs/1806.002877.
Denny Vrandecic and Markus Krötzsch. 2014. Wiki- data: a free collaborative knowledgebase. Commun.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of- the-art natural language processing. In Pro- ceedings of the 2020 Conference Empirical Meth- ods in Natural Language Processing: System Demon- strations, EMNLP 2020- Demos, Online, November 16-20, 2020. Association for Computational Linguis- tics.
ACM, 57(10).
Peifeng Wang, Filip llievski, Muhao Chen, and Xi- ang Ren. 2021. Do language models perform gen- eralizable commonsense inference? In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/UCCLLP 2021 of Findings of ACL. As- sociation for Computational Linguistics.
Weiqi Wang, Tianqing Fang, Baixuan Xu, Chun Yi Louis Bo, Yangqiu Song, and Lei Chen. 2023a. CAT: A contextualized conceptualization and instan- tiation framework for commonsense reasoning. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics ( (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023. Association for Computational Linguisties.
Wentao Wu, Hongsong Li, Haixun Wang. and Kenny Qili Zhu. 2012. Probase: a probabilistic tax- onomy for text understanding. In Proceedings of the ACM SIGMOD International Conference on Manage- ment of Data, SIGMOD 2012, Scottsdale, A7, USA, May 20-24, 2012. ACM.
William Yang Wang and Diyi Yang. 2015, That's so an- noying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic cat- egorization of annoying behaviors using #petpeeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal. September 17-21, 2015. The Association for Computational Linguis- tics.
Changlong Yu, Weiqi Wang, Xin Liu, Jiaxin Bai, Yangqiu Song, Zheng Li, Yifan Gao, Tianyu Cao, and Bing Yin. 2023. FolkScope: Intention knowledge graph construction for E-commerce commonsense discovery. In Findings of the Association for Com- putational Linguistics: ACI. 2023, Toronto, Canada. Association for Computational Linguisties.
Hongming Zhang, Xin Liu, Haojie Pan, Haowen Kc, Jiefu Ou Tianqing Fang. and Yangqiu Song. 2022. ASER: towards large-scale commonsense knowledge acquisition via higher-order selectional preference over eventualities. Artif. Intell., 309.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowd- hery, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May/ 1-5, 2023. Open
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NewrIPS.
Pei Zhou, Rahul Khanna, Seyeon Lee, Bill Yuchen Lin, Daniel Ho, Jay Pujara, and Xiang Ren. 2021. RICA: evaluating robust inference capabilities based commonsense axioms. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021. Association for Computational I Linguistics.
Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNT 2019, Hong Kong, China, Novem- ber 3-7, 2019. Association for Computational Lin- guistics.

---

Appendices
aNLI
CSQA
PIQA
SIQA
WG
#QA Pairs
1,532 1,221
1,838
1,954
1,267
A Benchmark Descriptions
#Options 2
5
2
3
In this section, we introduce more details regarding five evaluation benchmarks.
Table 4: Statistics on the number of QA pairs and the number of options for each question within each bench- mark's 's validation split.
Abductive NLI (aNLI) (Bhagavatula et al., 2020) a Natural Langauge Inference (NLI) bench- mark that aims to infer the most plausible expla- nation based on given causal situation. For each question sample, the model is required to choose the more plausible hypothesis out of two options that fit the beginning and end of story. This bench- mark evaluates the model's abductive reasoning ability, which typically requires commonsense rea- soning.
not be publicly available. Detailed statistics on the number of QA pairs and the number of options per question are reported Table 4.
B Additional Explanations and Analyses
In this section, we aim to cover additional details regarding the CSKB conceptualization in CAR (Ap- pendix B.1), implementations of our system (Ap- pendix B.4), baselines (Appendix B.2 and B.3), experiments using ATOMIC-10X (Appendix B.5), analyses (Appendix B.6, B.7, and B.8), and gener- alizability experiments (Appendix B.9) that are not covered in the body text due space constraints.
CommonsenseQA (CSQA) (Talmor et al., 2019) is a question -nsweerin benchmark that eval- uates broad range of commonsense aspects. Each sample contains question and five choices. The question and some choices are generated from sub- graphs of ConceptNet (Speer al., 2017) while crowdsourcing annotators also annotate some dis- tractors. This benchmark evaluates the model's concept-level commonsense reasoning ability.
B.1 Definitions and Statistics of CSKB Conceptualization
PhysicallQA (PIQA) (Bisk et al., 2020) is question-answeriig benchmark that requires the model to select the more plausible option out of two possible continuations given a common sce- nario that requires physical commonsense to in- fer. This benchmark evaluates the model's physical commonsense reasoning ability.
Conceptualization plays a crucial role in general- izable commonsense reasoning. Previous studies have demonstrated its potential in aiding common- sense inference modeling (Wang et al., 2023a) and commonsense knowledge graph construction (Yu et al., 2023; Zhang et al., 2022). In our paper, follow the definition of conceptualization pro- posed by He al. (2022) and Wang al. (2023a) in conceptualizing an instance within an event a concept: (1) Events: Each event represents a commonly observed occurrence that encompasses valuable subsequential or inferential commonsense knowledge. In AbstractATOMIC, the events are the head events of all triples in ATOMIC without a wildcard (L'). (2) Instances: Within each event, multiple instances have been identified with seman- tic parsing tools, representing specific components of the event that are worthy conceptualization. (3) Concepts: Concepts are the conceptualization of each instance. These concepts are thus extracted from Probase/WordNet and further validated by human annotators or critic filtering models.
SocialIQA (SIQA) (Sap et al., 2019b) is question cenestion-answering benchmark that requires rea- soning about social interactions. Each sample con- tains a context that is derived from ATOMIC (Sap et al., 2019a), question, and three choices. The questions are automatically generated using nine templates that correspond to the nine relations in ATOMIC, and the correct answers are crowd- sourced. This benchmark evaluates the model's reasoning ability for emotional and social common- sense in daily situations.
WinoGrande (WG) (Sakaguchi et al., 2021) is a pronoun resolution benchmark. Each sample con- tains an emphasized pronoun and a short context description. The model is asked to choose the cor- rect reference given two options. This benchmark evaluates the model's pronoun resolution ability, which is also part of commonsense knowledge.
For event e, which the head of an ATOMIC triple, an instance refers to either an entity within the event or the complete event itself. Multi- ple instances can exist within a single event, de-
In our experiments, we use the validation splits these benchmarks as the official testing sets may
noted as i,, b2, ..,.... in E e. A concept corre-

---

sponds to the conseptualization of an instance.
Di
D
Total
and multiple conceptualizations can be associ- ated with a single instance, as demonstrated by C1, 2), (11, 01, 3),...,, (i2, C2, I).... (In, Gn, 1):-. (In Cz: m.. For instance, consider the event "PersonX is drunk when exiting the bar." In this case, two instances can be identified: "Per- sonX is drunk when exiting the bar" and "bar." The conceptualization for the instance "PersonX is drunk when exiting the bar" may include "drunk" or "enjoyed," while the instance "bar" can be con- ceptualized as an "entertainment place" or a "fun place."
Unnq. event
7,196
15,165
15,388
#Unq. instance
7,935
20,843
21,493
(i1, C1, 1). c1,
#Unq. concept
20,036
20,367
31,227
Avg. #eoncepp/event
18.21
24.57
32.73
Avg. #concept/instance
16.51
17.88
23.43
Table 5: Statistics of conceptualizations used in CAR, as reported by Wang et al. (2023a). 12, stands for human- annotated conceptualizations and Dj are unlabeled con- ceptualizations. Unq stands for unique, and Avg refers to average.
obtained by replacing i E h with its conceptual- izations. This is because the process of concep- tualizing head event omits its context in (r, l). Thus, the last fold of data stores the plausibility of such abstract commonsense triples (her r,t), where human annotations are conducted verify part of the triples' plausibilities. addition, we adopt a semi-supervised instantiation discriminator, pro- vided by Wang et al. (2023a), to pseudo-label the unannotated triples. Another threshold, T 0.9, is set to filter out plausible abstract triples.
In this paper, we leverage the AbstractATOMIC dataset, provided by He et al. (2022), as primary source for conceptualizations. Abstrac- (ATOMIC is a benchmark for conceptualized commonsense knowledge that is built upon the ATOMIC dataset (Sap et al., 2019a). It contains three folds of data, each conditioned on the orig- inal commonsense knowledge triples (h, r, in ATOMIC.
In the first fold, He et al. (2022) identify all possible instances {ir, i2, -.. S h} in each ATOMIC head event, using syntactic pars- ing through a spaCy' parser and matching with five human-defined rules. It is important to note that, unlike traditional entity-level conceptualiza- tion benchmarks, the identified instance in Abstrac- (ATOMIC can also be the entire head event i = h. In the next fold, each identified instance is heuristically matched against Probase (Wu al., 2012) and WordNet (Miller, 1995) via Gloss- BERT (Huang et al., 2019) to find their correspond- ing conceptualization candidates. Human annota- tions are conducted to verify part of the plausibility of the conceptualization candidates. To pseudo- use
In the CAR framework, for every ATOMIC event h, we retrieve every instance i's plausible concep- tualizations {ci,1 C22, ... from all plausible con- ceptualizations derived in the second fold to serve as the distractor sampling constraint. We also aug- ment the original (h, r, t) triples with their plausi- ble (he, siblings from both gomman-nnnotater and pscudo-labeled triples, explained in the last fold. These knowledge triples are then synthesized into QA pairs using our proposed method train the model to perform general reasoning. Detailed statistics for the conceptualizations and abstract commonsense triples we finally obtained from the AbstractATOMIC dataset are reported in Table 5 and Table 6, respectively.
label unannotated conceptualizations, semi-supervised conceptualization discriminator provided by Wang et al. (2023a) and set a threshold T = 0.9 to filter out plausible conceptualiza- tions. Additionally, we utilize a GPT22bbsed (Rad- ford et al., 2019) generator, trained on the concate- nation of annotated and positively pseudo-labeled conceptualizations, to generate additional concep- tualizations for further expanding the size of the conceptualization bank.
B.2 Baseline Performances
For SMLM (Banerjee and Baral, 2020), we adopt the official implementation of Banerjee and Baral (2020), which employs the CSKB that exhibits the highest alignment with each task. Specifically, So- cialIQA uses ATOMIC, while Commonsense(AA uses ConceptNet. For STL-Adapter (Kim et al., 2022), only those trained on ATOMIC are used for comparison in the body text. In this paper, all baseline performances are solely based on their officially reported results in their respective papers. As noted in the Limitations section, previous re-
However, is worth noting that such conceptual- ization may not yield plausible abstract knowledge when (r, )) is connected back to he, where he is
hhttp:: //spacy.ip. io/

---

Relation
ATOMIC
D2
D
Grande (Sakaguchi al., 2021), and Common- sense Knowledge Base Population (Fang al., 2021b,a). In this study, we aim to benchmark Chat- GPT's zero-shot performance on five QA evalua- tion benchmarks used in our zero-shot common- sense QA task. Following (Robinson et al., 2022), we design and leverage batch of prompts, shown in Table 7, to probe ChatGPT's predictions. The prompt provides ChatGPT with question and its possible choices, along with natural lan- guage command to control the response action of ChatGPT. We then parse the generations using our meticulously designed rules, where punctuations and irrelevant wordings will be dropped, and the first choice-letter prediction will be identified as ChatGPT's answer. Specifically, if ChatGPT hesi- tates and cannot make a concrete prediction, it will be counted as a wrong answer. The benchmarking results are shown in Table We observe that Chat- GPT demonstrates superior performance compared to GPT3.5 (Ouyang et al., 2022) and excels in tasks such as CommonsenseQA (Talmor et al., 2019) and SocialIQA (Sap et al., 2019b), potentially due to the high frequency of their questions and answers in large text corpora. However, its performance the remaining three benchmarks is suboptimal, sug- gesting that they are more challenging and require more complex reasoning (Bai et al., 2023; Ding et al., 2023) and implicit commonsense knowledge to solve. This intriguing outcome warrants further investigation to determine the reasons behind it and explore methods to boost the LLM's abilities in these challenging benchmarks.
Xfffect
78,832
12,168
412,455
offfect
28,351
3,526
113,301
Want
101.249
15,312
177.745
WWnt
43,079
5,408
38,938
xReact
62,969
8,923
295.044
oReact
26,570
3,030
104,038
xNeed
74,272
11,733
378,442
AAttr 110,791
14,249
275,224
xIntent
45,490
6,848
234,948
Total
572,053
81,197
1,0
Table 6: Statistics abstract commonsense triples used in CAR, as reported by Wang al. (2023a). DJ stands for human-annotated triples and D are unla- beled triples.
search in this area has primarily focused on using four CSKBs, namely ATOMIC (Sap et al., 201 9a). ConceptNet (Speer et al., 2017), WordNet (Miller, 1995), and WikiData (Vrandecic and Krötzsch, 2014). In order to comprehensively benchmark framework's performance in the field of zero- shot commonsense QA, we compare our results on ATOMIC against baseline methods that use multiple CSKBs despite the unbalanced amount of knowledge in such a comparison. Table 11 presents a full comparison of our method with all existing baselines. Notably, for models based on RoBERTa-Large, our approach trained only abstract knowledge injected ATOMIC achieves second place in the leaderboard, falling only be- hind Kim et al. (2022) with four CSKBs. While this comparison may unfair due to the unbal- anced amount of knowledge, it provides a strong justification for the excellent performance of our system. Our D-BERTa-v3-Lare--bee based model still surpasses all baselines on average, indicating the necessity of leveraging a strong pre-trained lan- guage model.
Generally speaking, CAR and conceptualization own the advantage over the large language model in the following aspects: (1) Smaller Model Size: The CAR framework offers models that are signif- icantly smaller in scale compared to LLMs (0.2% of a standard 175 billion parameter GPT-3 model) while maintaining comparable performance in a zero-shot setting. Such size makes it more efficient in terms of training and deployment. In contrast, advanced prompting techniques used in LLMs re- quire extensive computational resources, making the conceptualizatiio-baass based model more versatile and accessible to researchers with limited access resources for deploying LLMs. (2) Broader Com- monsense Knowledge: Conceptualization provides a broader range commonsense knowledge com- pared to current CSKBs. Integrating this type knowledge into generative models has been shown
B.3 Benehmarking Large Language Models
We then discuss our method for benchmarking large language models on five commonsense QA benchmarks. The emergence of Large Language Models (LLMs), such as ChatGPT (OpenAI, 2022), has been the hot trend in recent NL.P research. Nu- merous studies have evaluated the capability of LLMs on various NLP downstream tasks. Among them, Qin et al. (2023); Chan al. (2023) have shown that ChatGPT can achieve competitive per- formance on commonsense reasoning tasks, such as Co (Talmor et al., 2019), Wino-

---

Task
Prompt
Gen
Premise: Jim decided to be rockstar.
Choice A: but didn't know how to play instrument. Jim signed for guitar lessons.
aNLI
A.
Choice B: Jim knew he would need have nickname. Jim signed up for guitar lessons.
Which one more likely happen, given the premise? Only answer A without any other word.
Question: He was at the gym trying to build muscle, what is it called that he trying to build musele on?
Choice A: body of animal
Choice B: arm
CSQA
Choice C: bodybuilder
C
Choice D: body of dog
Choice E: human body
Which choice is correct? Only answer A or B or C'or D or E without any other word.
Goal: To remove an avocado from the shell
Choice A: cut the avocado lengthwise, remove the pit, and scoop with spoon
PIQA
Choice B: cut the avocado width wisc, remove the pit, and scoop with spoon Which choice can achieve the goal? Only answer or B without any other word.
Question: Robin went to polls posted her ballot candidate she wanted. result, Robin wanted to:
Choice A: bomb
candidate
SIQA
C.
Choice B: attend rally
Choice C: go home.
Which choice is correct? Only answer or B or without any other word.
Question: Jessica enjoyed simple, basic
Betty,
Choice A: Jessica bored having quiet existence.
WG
A
Choice Betty bored having quiet existence.
Which choice correct? Only answer B without any other word.
Table Prompts used for evaluating GPT3.5 and ChatGPT, Gen. refers to the generated outputs from ChatGPT.
to enhance their performance in commonsense soning tasks (Wang al., 2023a). Such knowl- edge also be dynamically encoded in language models during inference time (Chen et al., 2023b). (3) Advanced Prompting of LLMs: Conceptual- ization data introduces the potential for a more advanced prompting framework of reasoning with LLMs. The process of conceptualization and the instantiation of knowledge play crucial role in reasoning. Thus, future work may consider intro- ducing the "chain of concept" reasoning process further advance the popular "chain of thought" reasoning paradigm (Wei al., 2022; Wang et al., 2023b).
2019) with learning rate of 7e-6 and a max quence length of 128 to accommodate QA pairs with different lengths. When evaluating the model on downstream commonsense QA benchmarks, a maximum sequence length 80 is used. We select the best checkpoint according to the highest accu- racy achieved on the synthetic validation QA set. Each experiment is repeated using different random seeds three times, and the average performance is reported. To overcome the limited GPU memory issue, we utilize gradient accumulation with gra- dient accumulated four steps before every descent, and each step calculates the gradient of eight data samples. The model is warmed up with 5% of total iterations and evaluated every 1000 global steps, while the margin for the marginal ranking loss is set to 1, in line with the choices made by Ma et al. (2021) and Kim al. (2022). The Hugging- face model code for our RoBERTa-L.an Large model is roberta-large, and the for our DeBERTa-v3- Large model is microsoft/deerrta-ss- All of our experiments are conducted on eight NVIDIA A100 GPUs, cach with 40G graphical memory. Training a RoBERTa-based model typi- cally requires 14G graphical memory, while train- ing DeBERTa-based models requires 30G graphi- cal memory.
B.4 Implementation Details
We present additional implementation details for building our system. For the pre-trained language models, We use PLMs from the Huggingface Trans- formers2 Library (Wolf et al., 2020) as the vanilla model checkpoints. Our system relies heavily the open-source code repository² provided by Ma al. (2021) for synthesizing QA pairs and training the QA models. To optimize the model, we em- ploy an A optimizer (Loshchilov and Hutter,
Phttps: hhtggs/g/agigiaacccecemm cor./oosss/raassorrees// Phttps: //tp:/bb.ttttt.te/ www.eayee.aaaaaeea 1atps/yeeee11ttAA--ssa--

---

Model
CSKB
Critic
a-NLI
CSQA
PIQA
SIQA
WG
Avg.
Backbone: RoBERTa-Large
RoBERTa-I. (MR)
ATOMIC
N/A
70.8
64.2
72.1
63.1
59.2
65.9
RoBERTa-L (MM))
ATM-10S
0.9
69.6
58.1
72.3
58.3
57.2
63.1
RoBERTa-L (MR)
ATM-10X
0.8
70.1
58.9
71.5
58.2
$7.7
63.3
RoBERTa-L (MR)
ATM-10X
0.7
70.8
59.4
72.1
58.5
58.3
63.8
RoBERTa-L (MR)
ATM-10X
0.5
68.7
56.8
71.7
58.4
60.1
63.1
RoBERTa-L (MR)
ATM-10X
0.0
70.7
58.3
71.7
58.2
57.5
63.3
RoBERTa-I.
ATMATM- ATM-
0.9
71.7
66.3
73.2
62.8
60.7
66.9
RoBERTa-L.
ATM ALM- TOX
0.8
71.8
66.0
73.2
61.7
59.5
66.4
RoBERTa-L (MR)
ATM
0.7
71.6
63.6
72.9
62.2
59.8
66.4
RoBERTa-L (MR)
ATM ATMATM- ATM--OX
0.5
72.0
72.9
62.0
60.5
66.6
RoBERTa-L (MR)
ATM ATM- IOX
0.0
71.6
66.3
73.3
62.9
61.0
67.0
CAR-RoBERTa-L (Ours)
ATOMIC
N/A
72.3
64.8
73.2
64.8
61.3
67.3
CAR-RoBERTa-L (Ours)
ATM
N/A
72.7
66.3
73.2
64.0
62.0
67.6
Backhone: DcBERTa-v3-L.n. 3-raaa
ASM
DcBERTa-v3-L. (MR)
ATOMIC
N/A
76.0
67.0
78.0
62.1
76.0
71.8
DeBERTa-v3-L. (MR)
ATM-10X
0.9
74.5
70.8
78.9
59.7
72.2
71.2
DcBERTa-v3-1. (MR)
ATM-10X
0.8
74.2
70.6
79.5
59.2
70.7
70.8
DeBERTa-v3-L (MR)
ATM- 10X
0.7
74.6
69.9
79.3
60.0
70.2
70.8
DcBERTa-v3-1. (MR)
ATM-10X
0.5
74.1
70.4
78.8
58.9
70.1
70.5
DeBERTa-v3-L (MR)
ATM-JOS
0.0
75.1
71.6
79.0
59.7
71.7
71.4
DeBERTa-v3-L (MR)
ATM ATMATM-IO
0.9
75.4
71.3
73.4
61.7
75.3
71.4
DeBERTa-v3-L (MR)
ATM ATM-
75.4
71.8
63.4
76.0
72.4
DeBERTa-v3-L. (MR)
ATM AIME TOX
0.7
74.9
71.2
77.4
61.8
76.2
72.3
DeBERTa-v3-L (MR)
ATM
0.5
74.8
71.2
77.1
61.7
75.7
72.1
DeBERTa-v3-L (MR)
ATM-IM--OO
0.0
76.2
75.8
75.8
72.3
CAR-DeBERTa<V3-L (Ours)
ATOMIC
N/A
78.9
67.2
63.8
78.1
73.3
CAR-DeBERTa-<3-L. (Ours)
ATM
N/A
79.6
69.3
78.6
64.0
78.2
73.9
Large Language Models
GPT-3.5 (text-davinci-00)
N/A
N/A
61.8
60.7
65.4
ChatGPT (gpt-3. 5-tturb))
N/A
69.3
74.5
75.1
69.5
62.8
70.2
Table 8: Zero-shot evaluation results (%) on five commonsense question answering benchmarks using different critic thresholds for filtering ATOMIC-10X. The best results are bold-faced, and the second-bes best ones are underlined. ATM stands for the ATOMIC with abstract commonsense knowledge injected. ATM-10X stands for using ATOMIC-10X (West et al., 2022) as the source CSKB D. ATM indicates the ATOMIC with sampled knowledge from ATOMIC-I0X injected. Critic indicates the lower bound for filtering knowledge from ATOMIC- 10X, which means that only knowledge with critic score above the threshold will be selected.
B.5 Experiments with ATOMIC-10X
its impact on zero-shot commonsense QA. Specifi- cally, we study the role of filtering the knowledge in ATOMIC-10X using multiple critic thresholds to acquire high-quality knowledge and improve model performance.
ATOMIC-1OX is a machine-generann corpus de- veloped by West et al. (2022) using the sym- bolic knowledge distillation framework. They employed selective distillation approach to ex- tract knowledge from large language models like GPT-3 (Brown et al., 2020) by prompting them with head events and commonsense relations from the ATOMIC datasct. The extracted knowl- edge was used to train a student model to gen- erate symbolic knowledge graphs evaluated us- ing a separate critic model. The resulting corpus, ATOMIC-10X, surpassed the human-gent numan-generated cor- pus ATOMIC2020 (Hwang al., 2021) in scale, accuracy, and diversity.
We utilize ATOMIC-10X in two distinct sce- narios. First, as discussed in Section 5.2, we di- rectly train our QA models on ATOMIC-10X with- out integrating other CSKBs, such as ATOMIC and AbstractATOMIC We source all questions, answers, and distractors from ATOMIC-1OX and follow its original train/dev/test partitions to di- vide the data. The lemmatized tokens of the head event, excluding commonly seen subjects, prepo- sitions, and stopwords, are used keywords for each piece of knowledge, and the original QA syn- thesis pipeline proposed by Ma et al. (2021) is ap- plied. To ensure the quality the knowledge from
In this section, we provide additional explana- tions regarding the usage of ATOMIC- 10X in our paper and conduct further experiments to explore

---

ATOMIC-10X, we multiple critic thresholds to filter the dataset accordingly. The QA models are trained using marginal ranking loss on four sub- sets of ATOMIC- 10X, each with a different critic threshold of 0.9, 0.8, 0.7, and 0.5, along with an additional model trained on the complete ATOMIC- 10X dataset. Finally, we evaluate these models on five commonsense QA benchmarks in zero-shot setting and report the results in Table 8. Specifi- cally, models trained solely on 10X us- ing critic thresholds of 0.7 (RoBERTa) and 0.0 (DeBERTa) for filtering are responsible for the re- sults reported in Table 1. We observe that even using high critic thresholds to filter the knowledge in ATOMIC- 10X, the model still fails to improve beyond marginal. Meanwhile, training the mod- els only on ATOMIC-10X fails to surpass training on ATOMIC, which indicates that the amount of knowledge is not the critical element to determin- ing the performance. Rather, should be the diver- sity and quality of knowledge, where the human- annotated knowledge from ATOMIC is superior to those machine machine-generated ones from ATOMIC. Nonetheless, none of the models outperform those trained on corppptualization-ugmeeee
the knowledge in ATOMIC-IOX, either for direct training or augmentation, occasionally improves the model's performance on a specific benchmark. However, it fails to boost the overall performance across all benchmarks on average, which is con- sidered closer metric for evaluating the gener- alizable reasoning ability of commonsense QA model. Thus, we come to the conclusion that ATOMIC- is inconsistently helpful in improv- ing the zero-shot commonsense QA performances, with most times failing to improve, while concep- tualization resolves such issues and can benefit the model across all benchmarks significantly. One po- tential reason is that ATOMIC- 10X main contain noise that are not benefitial to the task of zero-shot commonsense QA, as demonstrated by Deng et al. (2023).
B.6 Training Dynamic Definitions
Training dynamic, as proposed by Swayamdipta et al. (2020), refers to the analysis of a model's behavior on individual instances during training on large datasets. This analysis examines the model's confidence in predicting the true class of an in- stance and the variability of this confidence across epochs. To achieve this, multiple checkpoints are saved throughout a training epoch, and probability scores are derived for each data instance to calcu- late their training dynamics. By plotting the train- ing dynamics of all instances on a data map, in- stances can be categorized into three groups: easy- te-lear,, ambiguous, and hard-to-learn. For in- stance, consider a QA pair where a model consis- tently assigns a higher logit score to the correct answer than to other distractors across multiple checkpoints during an epoch. In this scenario, the model exhibits high confidence and low variability for that specific instance, suggesting that it is easy to learn. Conversely, instances with higher vari- ability are ambiguous to the model, and those with low confidence are difficult to learn. Experimental results by Swayamdipta et al. (2020) demonstrates that training the model with ambiguous data con- tributes the most to out-of-distribution generaliza- tion.
ATOMIC using our CAR framework, which further validates the strengths of CAR.
In the second scenario. discussed in Sec- tion 6.1, we utilize ATOMIC-10X as means of augmentation extend the original ATOMIC dataset. This is achieved by randomly select- ing a specific number of knowledge triples from ATOMIC-10X, cquivalent to the total number of plausible abstract commonsense knowledge in AbSIraC and merging them back into the original dataset. The triples in the resulting ATOMIC10X-
ATOMIC are then trans- formed into QA pairs and used to train our model following the original pipeline suggested by Ma et al. (2021). Similar to the previous scenario, we set four thresholds, namely 0.9, 0.8, 0.7, and 0.5, to filter the triples in ATOMIC-10S 10X for augmentation quality control. In this way, the QA pairs' distrac- tors can come from ATOMIC and ATOMIC-10X. The models are then trained and evaluated on five benchmarks. Their zcro-shot commonsense QA evaluation results are reported in Table 8, and the best model, trained using a critic threshold of 0.8 for filtering with DeBERTa-v3-large as the back- bone, is responsible for the results indicated in Table 2. Interestingly, we observe that leveraging
Inspired by this finding, research investi- gates the role of abstract commonsense knowledge within the training set and the effects of leverag- ing conceptualization. Since our QA model is trained with a marginal ranking loss, as described in Section 4.3, it does not output a probability

---

score but rather MLM score for each option. Thus, the definition of model's confidence pro- posed by Swayamdipta et al. (2020) does not fit into our problem definition. To address this, we re-define the calculation of confidence to align with the model's degree of certainty in predict- ing an instance as the true class. Formally, de- note n as the number of saved checkpoints dur- ing an epoch for computing their training dynam- ies and the list of m options in (Qi, Ai) pair as A, = [AA1, A.,,..., A with As, being the ground-trune truth answer (1 S S m)). We define the confidence of the model for such QA pair in Equation 3, where o is the sigmoid function and Sy. is the score of option Ai at checkpoint c.
Models
aNLI
CSQA
PIQA
SIQA
WG
CAR (RoBERTa)
72.7
66.3
73.2
64.0
62.0
o w/o CA
72.3
64.8
73.2
64.8
61.3
o w/o CCQS
71.5
67.3
72.1
61.8
62.7
CAR (DeBERTa)
79.6
69.3
78.6
64.0
78.2
w/o CA w/o CCQS
78.9
67.2
78.6
63.8
78.1
78.2
68.1
78.1
63.5
78.3
components of CAR. CA stands for Conceptualization Augmentation, and CCQS stands for Concept-Constraing QA Synthesis. The following five columns denote the accuracy (%) each benchmark.
Table 9: Ablation study
not facilitate the model's generalization. Thus, ab- stract commonsense knowledge benefits zero-shot commonsense QA better than ATOMIC- 10X by providing more ambiguous conceptual knowledge, which aids in making the model more generaliz- able.
m d
(S 1.d
S
C(Qi, A1)
0
(3)
72
m
c=
Intuitively, this equation averages the gap be- tween the ground-truth answer's score and the score of each distractor. A larger gap indicates a more confident model when choosing the answer. Variability aligns with the definition established by Swayamdipta et al. (2020). Specifically, it is calculated as the standard deviation of the score gap between the ground-truth answer and the dis- tractors relative to the level of confidence exhibited throughout entire epoch, shown in Equation 4.
We also plot the changes in training dynamics on different QA benchmarks, comparing models with and without the injection of abstract knowledge. The plots are shown in Figure 7. We observe that the inclusion of abstract commonsense knowledge significantly improves the models' confidence downstream QA entries. However, the impact on the trend of variability is unclear. Nevertheless, this improvement in average confidence provides strong evidence for the model's enhancement in these downstream QA benchmarks.
(9) Fi, C(Q,,
V(Q.,
(4) By revisiting the plots in Figure 4, we observe that the inclusion of abstract commonsense knowl- edge enhances the model's confidence and re- duces variability when encountering knowledge in ATOMIC. The introduction of conceptualiza- tion appears widen the differences between the model's predicted scores for the correct answer and those for the distractors. This suggests that the cor- rect answer is more likely to be selected, leading to an improved learning outcome. However, the introduction of knowledge from ATOMIC-10X re- sults in a reversed trend, indicating that it does not aid in better learning ATOMIC. Furthermore, we observe that abstract knowledge derived from con- ceptualizations is more ambiguous to the model in ATOMIC, which theoretically contributes more to out-of-domain generalization. Nonetheless, ATOMIC-IOX still contains some easy-to-learn knowledge that does
B.7 Ablation Study
Next, we study the ablation of different compo- nents in our CAR framework determine the im- pact of utilizing conceptualization through various techniques. There are two critical components that distinguish CAR from traditional zero-shot QA sys- tems (Ma et al., 2021):
Conceptualization Augmentation: Augmenting the original commonsense knowledge CSKB with its conceptualizations to derive abstract com- monsense knowledge. This knowledge is then syn- thesized into QA pairs, enabling the model to rea- son from a more generalized perspective. Without this component, abstract commonsense knowledge is not incorporated into the CSKB. Conceptualiza- tions still remain as constraints for assisting QA pair synthesis, resulting in an approach that is simi- lar to applying our proposed QA synthesis protocol directly to ATOMIC.
the concepuualzzation-aageeee

---

Consept-Constraineed QA Synthesis: Constrain- ing a question's distractors by ensuring that none of their head events share common keyword or con- ceptualization with the question's keywords and
2.5
Easy Difficult
Accuracy (%)
2.0
conceptualizations. this component is dropped, the constraint will be downgraded, and only no sharing of common keywords between the question and distractors will be restricted. This approach in- troduces abstract commonsense knowledge into the CSKB and uses the original distractor generation strategy for synthesizing QA pairs.
1.0
0.5
0.0
NNL
CSQA
SIQA
PIQA
WG
Avg.
Commonsense Question-Answenin Benehmarks
We then train two batches of QA models, using RoBERTa-Laage and GGBERTa-v3-Larre backbone, by sequentially dropping the com- ponents mentioned above one time. Their zero- shot performances on five commonsense QA bench- marks are reported in Table 9. From the results, is observed that both components play important roles in CAR, with CCQS being more effective average. This underscores the significance of climinating false negative distractors, and concep- tualization proves to be useful tool for achieving this objective improving the QA model's overall performance.
Figure 6: Comparison of accuracy improvement (%)
the
with/without conceptualizationaaaeee -aggetettttnam for two groups of QA entries across five benchmarks. Avg. stands for averaging across all benchmarks.
particularly for questions that deviate significantly from ATOMIC across multiple benchmarks. This indicates that augmenting ATOMIC with conceptu- alizations can improve the model's generalizability, particularly for questions that tend to be out-of- distribution, requiring more relevant knowledge answer correctly.
B.9 Generalization to Other CSKBs
B.8
The Effect of Conceptualization
While our work primarily experiments with the AbstractATOMIC dataset as the conceptualization source of ATOMIC, we also aim extend our framework other CSKBs for able evaluation. To address this, follow Ma et al. (2021) and explore the feasibility of transfer- ring our framework to the CWWV dataset, which comprises multiple CSKBs including Concept- Net (Speer et al., 2017), WordNet (Miller, 1995), and WikiData (Vrandecic and Krötzsch, 2014). To accomplish this, we train conceptualization gen- erator based on GPT2 (Radford et al., 2019) and utilize ChatGPT (OpenAI, 2022) as two flexible generative conceptualizers. The generated con- ceptualizations are then transformed into abstract knowledge and integrated into the CWWV dataset. This augmented dataset is employed to train a zero- shot commonsense QA reasoner using our pro- posed CAR framework. We present the experi- mental results and compare them with baselines in Table 10. Our observations reveal modest im- provement an average accuracy of 1% compared to all baselines and comparable performance GPT3.5. These results demonstrate the effective- ness of incorporating conceptualizations from other CSKBs. In future research, we suggest exploring
Lastly, we study the improvement the general- izability of our framework with the aid of concep- tualizations by examining the accuracy gains on questions with varying levels of semantic overlap with knowledge in ATOMIC's training split. To do so, we sort the questions in every benchmark by their average BERTScore (Zhang et al., 2020) between each individual question entry against the whole training set in the original ATOMIC. We then split the questions into two sets based on their BERTScores, with the lower BERTScore in- dicating a lower semantic overlap and a greater need for the model to generalize to answer the question. These questions are denoted "Diffi- cult." Conversely, we refer to questions with high BERTScores "Easy."
generaliz-
Then, we train two QA models following the pipeline proposed by Ma et al. (2021), with one ATOMIC
trained on concepuualization-auummee and the other on ATOMIC only. We evaluate their performance five commonsense QA bench- marks and compare the performance gains between two sets of questions each benchmark, as shown in Figure 6. Results demonstrate that incorporat- ing conceptualizations positively impacts accuracy,

---

Model
CSKB
a-NLI
CSQA
PIQA
SIQA
WG
Avg.
RoBERTa-L (MR) (Ma et al., 2021)
CWWV
70.0
67.9
72.0
54.8
59.4
64.8
MTI. (Kim et al., 2022)
CWWV
69.6
67.3
72.5
52.0
57.2
63.7
ZS-Fusion (Kim al., 2022)
CWWV
69.6
67.6
73.1
53.7
59.3
64.7
MAR-RoBERTa-L (Ours)
CWwV
71.6
68.4
73.0
55.4
60.6
65.8
GPT-3.5 (text-davinei-oe)
N/A
61.8
68.9
67.8
68.0
60.7
65.4
ChatGPT (gpt-3.5-turbo)
N/A
69.3
74.5
75.1
69.5
62.8
70.2
Table 10: Zero-shot evaluation results (%) on five commonsense question answering benchmarks by models trained the CWWV dataset. CWWVO refers to the augmented CWWV dataset using generated conceptualizations from a trained GPT2 generator and ChatGPT.
automatic construction methods for conceptualiza- tion resources in other CSKBs and investigating their potential benefits for general commonsense reasoning.
constraint, the newly sampled distractors all strong negatives, allowing the model to learn from such negative commonsense knowledge. This because the distractors are sourced from triples that are more likely to irrelevant the original triple's context and, thus, more likely to be truly negative distractors.
C
Case Study
this section, we present case studies to demon- stratc the effectiveness of CAR. First, we discuss cases that illustrate the power of conceptualization augmentation, shown in Table 12. By transform- ing triples into abstract commonsense knowledge, we can introduce more general knowledge into the CSKB and improve its coverage. Moreover, the newly introduced triples were missing from the original CSKB. For instance, conceptualizing "Per- sonX plays the games together" as an "entertain- ment activity" introduces higher-level knowledge that cannot be simply represented by the original triple. Additionally, by synthesizing both types of triples into QA pairs, the QA model can learn both types of knowledge, which can help the model perform more generalizable reasoning on out-of- distribution commonsense QA benchmarks.
Next, in Table 13, we present QA pairs con- sisting of false negative options generated using keyword constraints during their synthesis from both the original ATOMIC and ATOMIC-10X. We also demonstrate how our concept constraint re- solves this issue. Through these case studies, we observe that the original distractors may contain one or even both plausible options, which is sub- optimal when training a QA model. Specifically, for distractors sampled from ATOMIC-10X, we ob- serve that several distractors are vague and general (denoted as "?"), which can be plausible in many contexts. For example, in various triples, adjectives like "happy" and verb phrases such as "do it" easy to be plausible and do not serve as significant distractions. This is not desirable when training a QA model. However, by using conceptualizations

---

Commonsense
SocialIQA
PIQA
WinoGrande
aNLI
0.2
0.2
0.2
0.2
0.2
AConfidence
0.0
0.0
0.0
0.0
0.0
-0.2
-0.2
0.1
0.0 AVariability
0.1
0.1
0.0
0.1
0.1
0.0
0.1
0.1
0.0
0.1
0.1
0.0
0.1
AVariability
AVariability
AVariability
A Variabillity
Figure 7: The change of training dynamics on various commonsense QA benchmarks by DeBERTa-v3-Large model trained abstract commonsense knowledge injected ATOMIC (ours) compared with the one trained only ATOMIC (Ma et al., 2021).
Model
CSKB
a-NLI
CSQA
PIQA
SIQA
WG
Avg.
Random
50.0
20.0
50.0
33.3
50.0
40.7
Majority
50.8
20.9
50.5
33.6
50.4
41.2
GPT2-L (Radford et al., 2019)
56.5
41.4
68.9
44.6
53.2
52.9
RoBERTa-L (Liu et al., 2019)
65.5
45.0
67.6
47.3
57.5
56.6
DeBERTa-v3-L (He et al., 2023)
59.9
25,4
44.8
47.8
50.3
45.6
Self-talk (Shwartz et ul., 2020)
32.4
70.2
46.2
$4.7
COMET-DynGen (Bosselut al., 2021)
ATOMIC
50.1
SMLM (Banerjee and Baral, 2020)
65.3
38.8
48.5
Backbone: RoBERTa-Large 340M
RoBERTa-L (Vamilla) (Liu et ul., 2019)
65.5
45.0
67.6
47.3
57.5
56.6
MICO (Su et al., 2022)
ATOMIC
44.2
56.0
RoBERTa-L. (MR) (Ma ct al., 2021)
ATM
70.8
64.2
71.7
61.0
60.7
65.7
RoBERTa-I. (MR) (Ma al., 2021)
ATOMIC
70.8
64.2
72.1
63.1
59.2
65.9
RoBERTa-L (MR) (Ma et al., 2021)
CWWV
70.0
67.9
72.0
54.8
59.4
64,8
RoBERTa-I. (MR) (Ma ct al., 2021)
CSKG
70.5
67.4
72.4
63.2
60.9
66.8
STL-PLM (Kim et al., 2022)
ATOMIC
71.6
64.0
72.2
63.2
60.5
66.3
MTL al., 2022)
CWWV
69.6
67.3
72.5
$2.0
$7.2
63.7
MTL al., 2022)
CSKG
69.8
67.1
72.0
61.9
59.3
66.0
STL-Adapter (Kim et al., 2022)
ATOMIC
71.3
66.5
71.1
64.4
60.3
66.7
STL.-Adapeer (Kim et al., 2022)
CSKG
71.5
66.7
72.1
64.7
59.0
66.8
ZS-Fusion (Kim et al., 2022)
CWWV
69.6
67.6
73.1
53.7
59.5
64.7
7S-Fusion (Kim ct al., 2022)
CSKG
72.4
68.3
73.0
66.7
60.9
68.3
MKIF (Guan al., 2023)
CSKG
72.5
71.0
73.1
61.0
M.R-ROBERTa-L... (Ours)
ATOMIC
72.3
64.8
73.2
64.8
61.3
67.3
CAR-RoBERTa-L (Ours)
ATM
72.7
66.3
73.2
64.0
62.0
67.6
Backbone: DeBERTa-v3-Large 435M
DeBERTa-v3 (MR) (Ma al., 2021)
ATM
74.0
65.4
73.8
59.5
73.9
69.3
DeBERTa-v3 33LL (MR) (Ma et al., 2021)
ATOMIC
76.0
67.0
78.0
62.1
76.0
71.8
CAR-DeBERTH-v3-LL (Ours)
ATOMIC
78.9
67.2
78.6
63.8
78.1
73.3
CAR-DeBERTa.v33LL (Ours)
ATM
79.6
69.3
78.6
64.0
78.2
73.9
Large Language Models
GPT-3.5 (text-davinci-803) ChatGPT (gpt-3.5-turbo)
61.8
68.9
67.8
68.0
60.7
65.4
69.3
74.5
75.1
69.5
62.8
70.2
Supervised Learning & Human Performance
RoBERTA-L (Supervised)
85.6
78.5
79.2
76.6
79.3
79.8
DeBERTa-v3-L (Supervised)
89.0
82.1
84.5
80.1
84.1
84.0
Human Performance
91.4
88.9
94.9
86.9
94.1
91.2
Table 11: Zero-shot evaluation results (%) on five commonsense question answering benchmarks with baselines trained multiple CSKBs. The best results are bold-faced, and the second-ber best ones are underlined. ATMC stands for ATOMIC with abstract commonsense knowledge injected and ATM stands for ATOMIC-10X (West et al., 2022). All baseline results consistent with their original papers. CWWV refers the combination of ConceptNet (Speer et al., 2017), VisualGenome (Krishna et al.. 2017), WikiData (Vrandecic and Krötzsch, 2014), and WordNet (Miller, 1995). CSKG (llievski al., 2021) consists ATOMIC (Sap et al., 2019 9a) and CWWV.

---

Original Triple
Original Synthetic QA
Conceptualized Triple
Conceptualized Synthetic QA
PersonX looks cute, oWant, asks PersonX on a date.
Wynne looks cule. As a result, others wanted to?
PersonX
[prettyl. oWant, asks PersonX on date.
Wynne [prettyl. As a result, oth- wanted to? A: thank him.
A: thank him.
B+: ask Wynne on date.
B+: ask Wynne on date.
C: thank Wynne.
C: thank Wynne.
PersonX
Ray sets a new record. result, Ray wanted to? A: get to safety, accept prize.
PersonX [achievement). xWant, accept the prize.
Ray [achievement). As result, Ray wanted to?
sets new record, XWant, accept the prize.
A: get to safety.
Ba: accept the prize.
C: send the email.
C: send the email.
PersonX plays the games logether, xNeed, lind someone play with.
Logan plays the game together. Before, Logan needed to? know the framework. Be: find somcone play with.
[entertaiment activityl, xNeed, find someone play with.
Jentertaiment activiry]. Before, Logan needed to?
A: know the framework.
Be: find someone play with.
C: wash the clothes.
C: wash the clothes.
Table 12: Case study of conceptualized triples and their synthesized QA pairs. Given an original triple from ATOMIC, we conceptualize the triple by replacing instance with its [plausible conceptualization] to form conceptualized triple. The conceptualized triples are then synthesized into QA pairs using the same ground-trute answer and distractors, sampled for the original triple, train the QA model. indicates the ground-truth answer.
Question
Distractor Sampling Strategy
Distractor
E.Neg.
show off the food.
Keyword (ATOMIC)
take them home.
Jamie makes Alex's breakfast. As a result, Jamie wanted to? (eat with Alex.)
be good person. do it.
Keyword (ATOMIC-10X)
?
Keyword Concept
discuss the question. gel warm.
x
x
enjoyed.
Keyword (ATOMIC)
good.
Berkeley joins Aspen's party. As a result, others felt? (looked up to, admired.)
happy.
Keyword (ATOMIC-1OX)
good.
Keyword Concept
upset for having to give up their keys. sad.
x
x
have the knowledge. help with preparation.
Keyword (ATOMIC)
Cody builds Logan house. Before, Cody needed to? (have construction material.)
Keyword (ATOMIC-1OX)
start. cat well.
?
Keyword Concept
have been smoking weed. be with others.
x
x
decide to go. buy apple seeds.
Keyword (ATOMIC)
Ash is at the mall with West's friends. Before, Ash needed to? (drive his car)
prepared. study the situation.
?
Keyword (ATOMIC- 10X)
Keyword Concept
tries to do it but can't fail the test.
x
x
Table 13: Case study of the false negative options in the original QA synthesis and how our proposed conceptualizatioo-coo contstiiiionnnntriinttt resolves such an issue. The (ground truth answer) appended below each question. Keyword represents only using keywords as the constraint for sampling distractor, while Concept refers using both the keywords and conceptualizations, F.Neg. refers to whether the distractor is a false negative one or not.


Reviews:
[{"review_id": "9NQFOsjjEi", "reviewer": "Reviewer_qJZg", "summary": "", "questions": "As Large language models (e.g. GPT-3.5-turbo) can directly be used for zero-shot commonsense reasoning and the performance can be further promoted with tricks like chain-of-thoughts, what is the main advantage of building new conceptualization resources for zero-shot commonsense reasoning?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed feedback. The following paragraphs respond to the reason-to-reject together with the question.\n\n# Weak Novelty\n\nThank you for raising the issue of novelty. While existing frameworks, such as the conceptualization framework by He et al. (2022) and the data synthesis framework by Ma et al. (2021), have laid the foundation, our paper introduces novel contributions in two key areas. First, we present **a conceptualization-guided negative sampling strategy** that effectively reduces the occurrence of false-negative examples. Second, we propose **a new general data augmentation technique** based on conceptualization, accompanied by **comprehensive analyses of the data attribution of conceptualized data** for reliable usage.\n\nIn terms of the negative sampling strategy (Section 4.2), we go beyond conceptualization augmentation alone. We suggest utilizing conceptualization as an extended constraint for negative sampling, ensuring that the sampled negative options have no common keywords or conceptualizations with those mentioned in the question. This novel approach to negative sampling holds promise for its **applicability to other tasks that require negative sampling**, such as CSKB Completion (Malaviya et al., 2019) and Population (Fang et al., 2021).\n\nSecond, previous works on conceptualization have primarily focused on intrinsic evaluations using proprietary datasets without exploring their utility in downstream applications. Adapting conceptualization for practical downstream scenarios represents a significant and non-trivial advancement. By introducing conceptualization as **a general data augmentation tool**, we pave the way for new avenues of research. Notably, our augmented data demonstrate even greater generalizability compared to data distilled from GPT3 (ATOMIC-10X).\n\nIn addition, our work is strengthened, made more interpretable, and innovative by conducting a thorough analysis of the properties of conceptualization augmented data (Section 6). Rather than simply applying conceptualization, we extensively analyze the data attribution of conceptualized augmented data, considering factors such as diversity, false-negative rate, and training dynamics. \nWe demonstrate that conceptualization leads to the generation of more *ambiguous* (defined in Swayamdipta et al., 2020, a type of data useful for training) training examples, which significantly aids in the generalization of the model for performing commonsense reasoning. This observation is consistent with the findings in the training dynamics paper by Swayamdipta et al. (2020). For instance, we offer an explanation for why **conceptualization produces superior augmented data compared to that derived from GPT-3** (ATOMIC-10x). The reason lies in the ability of conceptualization to provide more ambiguous training signals, thereby enhancing the model's robustness.\n\n# Advantages of Conceptualization over Advanced LLM Prompting\n\nThank you for considering the comparison between conceptualization and advanced LLM prompting techniques, such as CoT reasoning. We believe that there are three distinct advantages that highlight the superiority of conceptualization and our system.\n\nFirstly, our framework offers models that are **significantly smaller in scale** while maintaining comparable performance to LLMs like ChatGPT. For instance, our trained model, which is based on DeBERTa-v3-large, consists of just 400 million parameters, representing a mere 0.2% of a standard 175 billion parameter GPT-3 model. Despite its smaller size, our model surpasses ChatGPT in a purely zero-shot setting, showcasing its efficiency in both training and deployment. In contrast, advanced prompting techniques require extensive computational resources for deployment and operation. This feature makes **our model, trained with conceptualization, more versatile and accessible to a broader range of researchers in NLP community** who may lack the necessary access or resources to extensively prompt LLMs.\n\nFurthermore, conceptualization provides a broader range of commonsense knowledge, which is currently lacking in current CSKBs. Wang et al. (2023) have previously demonstrated the advantages of integrating this type of knowledge into commonsense inference generation (COMET; Bosselut et al., 2019). It is evident from this research that conceptualization serves as a valuable resource for training more resilient and dependable generative models. Consequently, the development of conceptualization resources holds great potential for **enhancing the performance of large language models in commonsense reasoning**, such as fine-tuning ChatGPT using conceptualization data.\n\nLastly, we envision that **conceptualization data introduces the potential for a more advanced prompting of Language Model Models (LLMs)**. As we have emphasized previously, the process of conceptualization and instantiation of knowledge plays a crucial role in reasoning. By incorporating the \"chain of concept\" reasoning within the \"chain of thought\" paradigm, we can employ conceptualization as an additional guiding mechanism to prompt LLMs. Exploring this direction could bridge a significant gap in current understanding and yield promising future improvements in LLM performance across various downstream tasks.\n\n# References\n\nMalaviya, C., Bhagavatula, C., Bosselut, A., & Choi, Y. (2019). Commonsense Knowledge Base Completion with Structural and Semantic Context. AAAI Conference on Artificial Intelligence.\n\nFang, T., Wang, W., Choi, S., Hao, S., Zhang, H., Song, Y., & He, B. (2021). Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset. Conference on Empirical Methods in Natural Language Processing.\n\nSwayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N.A., & Choi, Y. (2020). Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics. Conference on Empirical Methods in Natural Language Processing.\n\nHe, M., Fang, T., Wang, W., & Song, Y. (2022). Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization. ArXiv, abs/2206.01532.\n\nMa, K., Ilievski, F., Francis, J.M., Bisk, Y., Nyberg, E., & Oltramari, A. (2020). Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering. AAAI Conference on Artificial Intelligence.\n\nBosselut, A., Rashkin, H., Sap, M., Malaviya, C., Celikyilmaz, A., & Choi, Y. (2019, July). COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 4762-4779).\n\nWang, W., Fang, T., Xu, B., Bo, C.Y., Song, Y., & Chen, L. (2023). CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning. Annual Meeting of the Association for Computational Linguistics.\n\n***We sincerely appreciate your valuable advice and hope that our response will assist you in raising your score. Thank you once again!***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work provides extensive results of experiments and analyses, which shows the advantage of CSKB conceptualization for zero-shot commonsense question answering. \n\n2. The result also brings some insight about the weakness of automatically generated commonsense knowledge bases.", "reasons_to_reject": "1. The novelty of the method is weak. This work hugely relies on previous work on the conceptualization-augmented CSKB resource. The conceptualization (Section 4.1) is actually done in previous work (He et al. (2022) and Wang et al. (2023)). And the synthesizing QA method (Section 4.2) is almost the same as Ma et al. (2021) except for extend the negative-sampling constrains to the augmented resource. Therefore, the real contribution of this paper is relatively weak.\n\nMutian He, et al. 2022. Acquiring and modelling abstract commonsense knowledge via conceptualization. \n\nWeiqi Wang, et al. 2023. CAT: A contextualized conceptualization and instantiation framework for commonsense reasoning.\n\nKaixin Ma, et al. 2021. Knowledge-driven data construction for zero-shot evaluation in commonsense question answering. ", "questions_to_authors": "As Large language models (e.g. GPT-3.5-turbo) can directly be used for zero-shot commonsense reasoning and the performance can be further promoted with tricks like chain-of-thoughts, what is the main advantage of building new conceptualization resources for zero-shot commonsense reasoning?", "paper_topic_and_main_contributions": "This paper proposes a new solution for zero-shot commonsense question answering task via commonsense knowledge bases (CSKB)-based data augmentation. It proposes to leverage conceptualization-augmented CSKB to enlarge the range of synthesized QA instances and reduce false-negative distractors. Experiments show that the overall framework performs better than previous zero-shot baselines, and the conceptualization-augmented CSKB is a better than a CKSB distilled from GPT-3 as the source for learning general reasoning. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6JSMUrPWUL", "reviewer": "Reviewer_NWmS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed feedback. The following respond to the reasons-to-reject.\n\n# The Necessity and Importance of Multi-Choice Question Answering (MCQA)\n\nFirst, the inclusion of negative options is essential as it acknowledges **the subjective and comparative nature of commonsense knowledge**. Rather than assuming the question-answer pair as an absolute truth, **Multiple-Choice Question Answering (MCQA) considers the choices as plausible, with minimal cost of incorporating bridging inferences in the causal field** (COPA; Roemmele et al., 2011).\n\nFor instance, in certain scenarios, even negative options can be deemed plausible when presented with counterfactual context. For example, in zero gravity, an apple may float in the air instead of falling down. However, when answering a commonsense reasoning question, it is prudent to compare it against the most common case without imposing additional restrictions. Hence, when considering the question of what happens when an apple ripens, selecting \"apples fall when ripe\" as a plausible choice is more appropriate than choosing \"apples float in the air when ripe.\" To this extent, in order to evaluate a commonsense reasoning model's ability to compare and distinguish the most plausible knowledge in typical scenarios that are commonly recognized, it is **beneficial to incorporate negative options as comparative benchmarks to form an efficient evaluation** approach.\n\nSecond, practically speaking, **MCQA serves as the predominant representation for commonsense reasoning datasets, encompassing a wide range of benchmarks that explore various aspects of commonsense**. These include SWAG, HellaSWAG, CommonsenseQA (v1.0, v2.0), SocialIQA, PhysicalIQA, ARC, OpenbookQA, CosmosQA, WinoGrande, COPA, CODAH, MC-TACO, a-NLI, among others (citations omitted). It is highly debatable to argue that the previous constructions and subsequent works in these commonsense QA datasets have become obsolete, as solving these tasks remains **a crucial objective even for large language models**.\n\n# Impact of Pre-training and Knowledge Injection\n\nThank you for your attention to the impact of pre-training. We would like to provide further clarification on this approach, which involves injecting knowledge into a model that is similar to transfer learning. By using synthetic question-answering pairs from commonsense knowledge bases, we enable **the model to become familiar with specific types of question-answering data and grasp a wide range of commonsense knowledge encapsulated within the CSKBs and conceptualized CSKBs**. This method effectively **transfers commonsense knowledge into the model**, as demonstrated in studies by Ma et al. (2021), Bosselut et al. (2021), and Shwartz et al. (2020) (self-talk). Notably, in our experiments, models perform exceptionally well on SocialIQA, a commonsense QA dataset derived from ATOMIC, the CSKB used for pre-training. This further demonstrates the model's ability to transfer knowledge from CSKBs to answer downstream QA tasks.\n\nIt is also important to highlight that the synthetic QA pairs used in pre-training are **not limited to head, entity, and relation only**. For instance, QA pairs derived from ATOMIC predominantly feature an event in the question and an adjective/event in the answer. The objective of this pre-training approach involves comparing the loss between the correct choice and distractors, thereby **training the model to better discriminate plausible commonsense assertions and knowledge**. Such pre-training enables the model to effectively differentiate among diversified scenarios and identify the most plausible commonsense option. Additionally, datasets involving longer contextual questions or multi-hop reasoning, such as PIQA and SIQA, also benefit from this type of pre-training. As shown by our experiments, **the model’s improved sensitivity also generalizes well to these complex reasoning scenarios**. \n\n# References\n\nRoemmele, M., Bejan, C. A., & Gordon, A. S. (2011, March). Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series.\n\nMa, K., Ilievski, F., Francis, J.M., Bisk, Y., Nyberg, E., & Oltramari, A. (2020). Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering. AAAI Conference on Artificial Intelligence.\n\nBosselut, A., Le Bras, R., & Choi, Y. (2021, May). Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering. In Proceedings of the AAAI conference on Artificial Intelligence (Vol. 35, No. 6, pp. 4923-4931).\n\nShwartz, V., West, P., Le Bras, R., Bhagavatula, C., & Choi, Y. (2020, November). Unsupervised Commonsense Question Answering with Self-Talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 4615-4629).\n\n***We sincerely appreciate your valuable advice and hope that our response will assist you in raising your score. Thank you once again!***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe proposed idea, i.e., using knowledge abstraction to improve coverage and reduce false negatives, is sound and novel.\n2.\tThe proposed method brings significant performance gain without direct supervision.\n3.\tThe paper is well-organized and easy to follow.", "reasons_to_reject": "1.\tI understand that a multi-choice QA setting would facilitate evaluation. But I do not think it still makes sense today to assume there exist options to choose from when conducting commonsense reasoning.\n2.\tIt is unclear why training LMs on synthetic QA pairs based on knowledge triplets can lead to downstream performance gain. Tasks like PIQA and SIQA have long context which is not simply about a head entity and relation. Also, certain questions require multi-hop reasoning. It would be helpful if the work can provide some explanations on the impact of pre-training.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper aims to solve zero-shot commonsense QA tasks. It constructs synthetic QA pairs based on existing knowledge triplets for pre-training a masked language model. To improve coverage and reduce false negatives, the work adopts knowledge abstraction. This helps: 1) augment the commonsense KG to an abstract one and 2) filter false negatives (tail entities). During inference, the trained LM is directly applied to conduct QA without further tuning. Experiments show effectiveness of the proposed method on a wide range of commonsense QA tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NSXRX8uM6M", "reviewer": "Reviewer_hZ2z", "summary": "", "questions": "1. I have trouble understanding the difference between event vs instance vs concept, in table 4. Does event refers to the head of a triple, and instances refer to all heads in KG that share the same meaning, and concepts refer to all abstraction of the current head? It would be good to define these terms somewhere. \n2. Have you tried to compute the human accuracy on the synthetic data? In Ma et al. 21, human accuracy is only around 80%, which is also a sign of false negatives in data. Would human accuracy be higher on your dataset? \n3. Have you studied the impact of the size of the training set? If you downsample the augmented data to the original ATOMIC size (same as Ma et al. 21), what effect would it have on the model’s performance?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed feedback. The following paragraphs respond to the reasons-to-rejects together with the questions.\n\n# Generalization to Other CSKBs\n\nThank you for addressing the issue of generalizability. In fact, there exist systematic approaches, other than large-scale annotation, to acquire conceptualization data for other CSKBs. \n\nFor example, in our paper, we leveraged **a GPT2 conceptualization generator that was trained on AbstractATOMIC**. This generator is capable of generating conceptualizations for specific instances by considering the instance itself and its source event (*refer to our definitions below*). This powerful tool can be effectively utilized to **generate conceptualized triples for CSKBs beyond ATOMIC**. Furthermore, we recently discovered that **ChatGPT exhibits robust conceptualization capabilities**. By using few-shot prompts synthesized from AbstractATOMIC, we can **prompt ChatGPT to generate conceptualized triples based on any given knowledge in other CSKBs**. \n\nTo address your concern, we have chosen **CWWV** (including ConceptNet, WordNet, and WikiData) as an additional dataset, distinct from ATOMIC, which was utilized by Ma et al. (2021) in their work. We employ the **GPT2 generator and ChatGPT to conceptualize the triples in CWWV**. Due to insufficient time during the response period, we are unable to conduct large-scale annotations for verification and evaluation. Thus, we prepared a set of case studies for your review.\n\n| Event                                                              | Instance                                       | GPT2 Conceptualization | ChatGPT Conceptualization |\n|--------------------------------------------------------------------|------------------------------------------------|------------------------|---------------------------|\n| You are likely to find a basketball court in high school gymnasium | basketball court                               | sport venue            | sport arena               |\n| You are likely to find a basketball court in high school gymnasium | high school gymnasium                          | gym                    | gym                       |\n| You are likely to find an electric socket in electrician's truck   | electric socket                                | plugs                  | Power sockets             |\n| electric power distribution grid is a facility                     | electric power distribution grid is a facility | facility               | base facility             |\n| An activity pilots can do is fly a jet                             | pilot                                          | driver                 | guidance                  |\n| An activity pilots can do is fly a jet                             | fly a jet                                      | control                | navigation                |\n| a pillow case is for holding items                                 | pillow case                                    | cover                  | pillow protector          |\n\nThese case studies demonstrate that **small generative models, when fine-tuned on conceptualization data, and large language models can both be effective in conceptualizing other CSKBs**. Notably, large language models can achieve this without relying on AbstractATOMIC. We then **incorporate these conceptualized triples, referred to as AbstractCWWV, into the traditional QA synthesis** pipeline introduced by Ma et al. (2021). **A QA model, based on RoBERTa-Large, is trained on the QA pairs synthesized from CWWV + AbstractCWWV directly using the random distractor sampling strategy**. The results from the evaluation on five QA benchmarks, in comparison with the baselines, are presented below.\n\n| Method            | CSKB                | aNLI | CSQA | PIQA | SIQA | WG   | Avg. |\n|-------------------|---------------------|------|------|------|------|------|------|\n| Ma et al. (2021)  | CWWV                | 70.0 | 67.9 | 72.0 | 54.8 | 59.4 | 64.8 |\n| Kim et al. (2022) | CWWV                | 69.6 | 67.6 | 73.1 | 53.7 | 59.5 | 64.7 |\n| **Our RoBERTa-Large** | CWWV + AbstractCWWV | 71.6 | 68.4 | 73.0 | 55.4 | 60.6 | **65.8** |\n\nDue to time constraints during the response period, we were unable to reproduce CAR with CWWV+AbstractCWWV, and there was a lack of a critic model or human evaluation for filtering the generations. As a result, **we strongly believe that the current model and results are not fully optimized**. However, we did observe a **modest improvement of approximately 1% in accuracy compared with both baselines**. Therefore, **we are confident that there are potential avenues for extending our framework to other CSKBs with ease**. For instance, leveraging a fine-tuned GPT2 or other large language models. This would eliminate the need for the AbstractATOMIC dataset, as alternative resources can be constructed for other CSKBs automatically. By obtaining these resources, **our framework can be directly applied to other CSKBs, enabling the training of more robust and knowledgeable QA models**.\n\nIn addition, future research can explore conceptualizations distillations from large language models. The constructed resource can be utilized to benefit downstream tasks, such as COMET (Bosselut et al., 2019), and enable zero-shot commonsense QA.\n\n# Unclear Experiment Details\n\nWe are sorry that some details aren’t made clear right now.\n\n(a) To conduct the expert evaluation, **we carefully selected three annotators who possess considerable research experience in the field of commonsense**. These annotators are either undergraduate or graduate students and have actively participated in the publication of research papers. Prior to the evaluation, we provided them with thorough instructions on the task of conceptualization and all related annotation tasks. Moreover, they underwent pilot rounds to ensure their annotation abilities, and they achieved a minimum accuracy of 80% on a meticulously curated gold set comprising 20 annotation questions, handpicked by the authors of this paper. Subsequently, they were provided with comprehensive explanations elucidating the underlying rationales for any incorrect responses they provided during the pilot rounds. This process ensured that they possess a clear understanding of their mistakes and the associated reasoning behind them.\n\nOverall, our annotators demonstrated a high level of agreement among themselves, **with an inner annotation agreement (IAA) of 83% in terms of pairwise agreement. The Fleiss Kappa score, which measures inter-rater reliability, was calculated to be 0.64**. This score is comparable to the Fleiss Kappa score (0.62) reported by Ma et al. (2021) and **reflects substantial agreement among the annotators**. Consequently, we are confident in the **reliability and validity of our evaluations**.\n\n(b) The ATOMIC-10X augmentation undergoes filtering based on the critical value and is then sampled to ensure fairness among different augmentation methods. This process guarantees that the augmented knowledge aligns with the conceptualization in terms of quantity. Although the randomness introduced by this sampling might raise concerns regarding diversity and coverage, **we argue that it does not have a substantial impact**. This is primarily due to the **relatively large size of our sampled subset, which constitutes approximately 20% of the complete ATOMIC-10X dataset**. Consequently, each random sample contains a diverse range of knowledge that maintains a comparable level of diversity and coverage to the original ATOMIC dataset. \n\nTo further validate this assertion, we conducted three additional sampling iterations and calculated the automatic diversity measure (insufficient time to re-conduct expert evaluation three times). **The resulting diversities were 33.8%, 36.4%, and 34.4%, averaging at 34.9%, which closely aligns with the diversity reported in our paper (35.6%).** Hence, we are confident that the **random sampling approach does not yield significant differences in terms of diversity and coverage** when compared to ATOMIC.\n\n# Event vs. Instance vs. Concept\n\nWe base our approach on the definitions proposed by He et al. (2022) and Wang et al. (2023) for formulating conceptualizations of commonsense knowledge. \n\n**Events:** Each event represents a commonly observed occurrence that encompasses valuable subsequential or inferential commonsense knowledge. In AbstractATOMIC, the events are the head events of all triples in ATOMIC without a wildcard (‘_’). \n\n**Instances:** Within each event, multiple *instances* have been identified with semantic parsing tools, representing specific components of the event that are worthy of conceptualization. \n\n**Concepts:** Concepts are the conceptualization of each instance. These concepts are thus extracted from Probase/WordNet and further validated by human annotators or critic filtering models. \n\nTo summarize, the following definitions can be established: For an event $e$, which is the head of an ATOMIC triple, an instance refers to either an entity within the event or the complete event itself. Multiple instances can exist within a single event, denoted as ${i_1, i_2, i_3, …, i_n} \\in e$. A concept corresponds to the conceptualization of an instance, and multiple conceptualizations can be associated with a single instance, as demonstrated by $(i_1, c_1,1), (i_1, c_1,2), (i_1, c_1,3),..., (i_2, c_2,1), ..., (i_n, c_n,1), …, (i_n, c_n,m)$.\n\nFor instance, consider the event \"PersonX is drunk when exiting the bar.\" In this case, two instances can be identified: \"PersonX is drunk when exiting the bar\" and \"bar.\" The conceptualization for the instance \"PersonX is drunk when exiting the bar\" may include \"drunk\" or \"enjoyed,\" while the instance \"bar\" can be conceptualized as an \"entertainment place\" or a \"fun place.\"\n\n# Synthetic QA Accuracy/Plausibility\n\nWe have also conducted expert annotations of the synthetic QA set to determine the synthetic QA pairs’ accuracies. We only select QA pairs that are synthesized from triples in the original ATOMIC for fair comparison. The results indicate that the expert annotator selects the correct answer for the question with an accuracy of approximately **86%** among 300 sampled QA pairs. This accuracy **surpasses that of the synthesis (78%) conducted by Ma et al. (2021)**. \n\nDespite this improvement, we did not include this information in our paper. The reason for this omission is that the accuracy is actually limited (more precisely, upper bounded) by the plausible rate of knowledge in ATOMIC, which is hard to quantify exactly. Given that certain knowledge within ATOMIC is challenging to comprehend, the corresponding QA pairs generated from this knowledge tend to be difficult to answer correctly. Ma et al. (2021) reported this issue as “Grammarly Incorrect,” which we also observed in our annotation. Consequently, we have opted to employ the **false-negative ratio as the main evaluation metric** to reflect this issue. This offers a more direct assessment and remains **unaffected by implausible or ungrammatical knowledge in ATOMIC**. Our expert evaluations demonstrate that **our proposed method effectively minimizes the likelihood of generating false-negative distractors**.\n\n# Impact of Training Data Size\n\nThank you for raising the question regarding the size of training data. To answer your question, **we add experiments by downsampling the augmented training data (synthetic QA pairs from both ATOMIC and AbstractATOMIC) to the number of QA pairs synthesized only from ATOMIC**. DeBERTa-v3-Large is used as the backbone, and the results are reported in the table below:\n\n| Method                              | CSKB                  | aNLI | CSQA | PIQA | SIQA | WG   | Avg. |\n|-------------------------------------|-----------------------|------|------|------|------|------|------|\n| Baseline (Ma et al., 2021)          | ATOMIC                | 76.0 | 67.0 | 78.0 | 62.1 | 76.0 | 71.8 |\n| CAR                                 | ATOMIC                | 78.9 | 67.2 | 78.6 | 63.8 | 78.1 | 73.3 |\n| CAR **(Downsample to Half ATOMIC)** | ATOMIC+AbstractATOMIC | 77.3 | 64.0 | 75.6 | 61.0 | 76.4 | 70.9 |\n| CAR **(Downsample to ATOMIC)**      | ATOMIC+AbstractATOMIC | 79.2 | 67.7 | 77.0 | 61.9 | 77.5 | 72.7 |\n| CAR **(Without Downsampling)**      | ATOMIC+AbstractATOMIC | 78.5 | 69.3 | 78.1 | 63.9 | 78.1 | 73.6 |\n\nUpon downsampling the synthesized QA pairs obtained from a combination of ATOMIC and AbstractATOMIC to match the quantity of those derived solely from ATOMIC, we observed an average performance decrease of 0.9%. This decline resulted in the model's performance falling below that of CAR, which operates solely on ATOMIC without any conceptualization augmentation. Moreover, when downsampling to even half the quantity of ATOMIC QA pairs, the performance dropped significantly, falling below the baseline established by Ma et al. (2021).\n\nGiven that conceptualization augmentation introduces a substantial amount of abstract commonsense knowledge, such downsampling leads to the loss of a considerable portion of both abstract knowledge (from AbstractATOMIC) and concrete knowledge (from ATOMIC). Therefore, we can conclude that **both types of knowledge are crucial for developing a robust zero-shot commonsense QA model**. While augmenting ATOMIC with conceptualization proves advantageous to the model, **it still necessitates a substantial amount of data for the model to exhibit generalizability and optimal performance**.\n\n# References\n\nHe, M., Fang, T., Wang, W., & Song, Y. (2022). Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization. ArXiv, abs/2206.01532.\n\nMa, K., Ilievski, F., Francis, J.M., Bisk, Y., Nyberg, E., & Oltramari, A. (2020). Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering. AAAI Conference on Artificial Intelligence.\n\nWang, W., Fang, T., Xu, B., Bo, C.Y., Song, Y., & Chen, L. (2023). CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning. Annual Meeting of the Association for Computational Linguistics.\n\nBosselut, A., Rashkin, H., Sap, M., Malaviya, C., Celikyilmaz, A., & Choi, Y. (2019, July). COMET: Commonsense Transformers for Automatic Knowledge Graph Construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 4762-4779).\n\n***We assure you that we will include all the mentioned content in our final camera-ready version to improve the clarity and readability of our paper. We sincerely appreciate your valuable advice and hope that our response will assist you in raising your score. Thank you once again!***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is well-motivated and very intuitive. The issue of false negatives in synthetic data is very challenging to solve. By abstracting the higher-level concept out of the knowledge triple surface form, the false negatives can be more effectively detected and filtered out.  \n2. The results show that the model’s performance improved significantly by training on the newly generated synthetic data (for both models and on all 5 tasks).", "reasons_to_reject": "1. The generalization of this method to other knowledge graphs is unclear. The most important step in the proposed pipeline, the conceptualization of the knowledge triples, requires the AbstractATOMIC dataset (He et al. 22). It would be interesting to see if there are any alternatives to get conceptualization of the knowledge triple without AbstractATOMIC, or if the proposed method works on other knowledge graphs such as ConceptNet. \n2. Some experimental details are not well-explained, which might weaken the arguments of this paper. For example, (a) In section 6.1, human evaluations are used to understand the diversity of the augmented knowledge triples, I’m wondering how many annotators labeled each instance, and is the inter-annotator agreement computed? (b) In lines 482-483, the knowledge is randomly sampled from ATOMIC-10X, then in lines 493 - 494, these sampled knowledge are treated as augmentation when computing the diversity metric. In this case, the comparison in terms of diversity and coverage might be unfair because different random samples of the data could lead to very different coverage of knowledge? \n3. For some minor problems, please see questions", "questions_to_authors": "1. I have trouble understanding the difference between event vs instance vs concept, in table 4. Does event refers to the head of a triple, and instances refer to all heads in KG that share the same meaning, and concepts refer to all abstraction of the current head? It would be good to define these terms somewhere. \n2. Have you tried to compute the human accuracy on the synthetic data? In Ma et al. 21, human accuracy is only around 80%, which is also a sign of false negatives in data. Would human accuracy be higher on your dataset? \n3. Have you studied the impact of the size of the training set? If you downsample the augmented data to the original ATOMIC size (same as Ma et al. 21), what effect would it have on the model’s performance?", "paper_topic_and_main_contributions": "This paper proposed a new method for constructing synthetic data for zero-shot commonsense question answering. The previous method for populating synthetic QA pairs from commonsense knowledge graphs results in a large amount of false negatives. Consequently, training models on those data leads to sub-optimal performance. The authors propose to first get the conceptualized knowledge triple (getting the abstraction of the head concept/event while keeping the relation and tail the same). Then they followed the same synthetic QA generation pipeline, but the sampled distractors are filtered based on both the triple itself and the conceptualization of the triple (through keyword matching). In this way, the number of false negatives in the distractor can be significantly reduced. The authors conducted experiments by training RoBERTa and DeBERTa models on the resulting synthetic QA sets and evaluating on 5 commonsense QA benchmarks in a zero-shot fashion. The results show that the proposed method outperforms various baselines on all 5 tasks.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new solution to gather negative synthetic data to train an LM for zero-shot commonsense QA. For this, they use conceptualization/ abstraction over Commonsense KB triples to enhance knowledge coverage and in guiding an effective negative sampling strategy to mitigate false-negatives in the generated synthetic data. Their method gets empirical gains over data distilled directly from LLMs and they provide insightful analysis for on-the-fly generated CKB. This is a useful framework with good results, backed by good experimental analysis.

It would be interesting to see the generality of their conceptualization based pretrained LM by studying if it applies to commonsense generation tasks such as script generation (it should be within the scope as they use ATOMIC). For generalization of the approach w.r.t. other KBs, please include the new results in the camera ready. It would also be interesting to see how conceptualization can be used to guide LLM prompts.

In summary, this is a useful data augmentation technique that can yield good results on other tasks in the future (remains to be seen).
Decision: Accept-Findings