CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning
Year: 2023

Abstract:
Commonsense reasoning, aiming at endowing machines with a human-like ability to make situational presumptions, is extremely challenging to generalize. For someone who barely knows about meditation, while is knowledgeable about singing, he can still infer that meditation makes people relaxed from the existing knowledge that singing makes people relaxed by first conceptualizing singing as a relaxing event and then instantiating that event to meditation. This process, known as conceptual induction and deduction, is fundamental to commonsense reasoning while lacking both labeled data and methodologies to enhance commonsense modeling. To fill such a research gap, we propose CAT (Contextualized ConceptuAlization and InsTantiation), a semi-supervised learning framework that integrates event conceptualization and instantiation to conceptualize commonsense knowledge bases at scale. Extensive experiments show that our framework achieves state-of-the-art performances on two conceptualization tasks, and the acquired abstract commonsense knowledge can significantly improve commonsense inference modeling. Our code, data, and fine-tuned models are publicly available at https://github.com/HKUSTKnowComp/CAT.

heading: 1 Introduction
text: “Concepts are the glue that holds our mental world together.”– Murphy (2004)
Commonsense reasoning is a crucial ability for machines to make situational presumptions and draw inferences from the knowledge that reflects our humans’ understanding of situations and common facts (Davis, 1990; Davis and Marcus, 2015). It has gained increasing popularity in the Natural Language Processing (NLP) community with the emergence of CommonSense Knowledge Bases (CSKB) (Sap et al., 2019a; Speer et al., 2017;
∗ Equal Contribution
PersonX watches football game, as a result, PersonX will:
feel relaxed
PersonX plays with his dog, as a result, PersonX will: be happy and relaxed
PersonX [observe] as a result, PersonX will:
feel relaxed
PersonX [relaxing event] as a result, PersonX will:
feel relaxed
Conceptualization (watches football game → relaxing event)
Instantiation (relaxing event → plays with his dog)
Abstract Knowledge
Aided Reasoning
Conceptualization (watches football game
→ observe)
Wrong Abstract Knowledge
Figure 1: A demonstration of commonsense reasoning on an unknown situation, PersonX plays with his dog, with the aid of abstract commonsense knowledge. Decontextualized conceptualization, such as observe, may yield wrong abstract commonsense knowledge that cannot be instantiated within the corresponding context.
Hwang et al., 2021) and large language models (Bosselut et al., 2019; Rajani et al., 2019; Liu et al., 2022b; Su et al., 2022; Yu et al., 2022b). However, when encountering situations beyond the data given, more abstract background knowledge must be acquired and generalized to assist the reasoning (Tenenbaum et al., 2011), and language models trained with an autoregressive language modeling objective do not explicitly leverage such abstract knowledge during inference.
Instead, humans rely on conceptual induction and deduction (Murphy, 2004) to make inferences on novel situations without the need to memorize all special cases. As shown in Figure 1, humans can derive conceptualizations based on the assertion that “PersonX watches a football game, as a result, he feels relaxed” to infer that “relaxing events can make someone feel relaxed,” where the acquired abstract commonsense knowledge can be further used as general knowledge to perform reasoning on similar or associated situations. A new commonsense knowledge “PersonX plays with his dog, as a result, he feels happy and relaxed” can be deduced by instantiating relaxing events to playing with his dog.
13111
As the cornerstone of generalizable commonsense reasoning, such a process is extremely challenging for machines to replicate due to the absence of contextualized conceptualizations and abstract commonsense knowledge in CSKBs and a lack of relevant methodologies.
Yet, existing works address the process of induction and deduction separately via conceptualization and instantiation. Several methods performing conceptualization are proposed with a specific focus on entity-level (Durme et al., 2009; Song et al., 2011; Gong et al., 2016; He et al., 2020; Peng et al., 2022; Song et al., 2015) and event-level (Chen et al., 2020; He et al., 2022) semantics. Instantiation (Allaway et al., 2023), as the process that simulates conceptual deduction, is tackled separately and not leveraged by these methods. Though abstract commonsense knowledge can be derived by using existing conceptualization methods to abstract a certain instance from factual commonsense knowledge, several limitations still exist.
First, the plausibility of abstract commonsense knowledge banks on both the correctness of conceptualization and proper contextualization under specific assertions. The latter one, which is an essential step for the deduction of abstract knowledge, is missing from current methodologies. Take Figure 1 as an example, the concept observe will not necessarily lead to the result of “feeling relaxed”, as observe omits the entertaining property of the original instance as a cost of abstraction. Second, instantiating abstract commonsense knowledge can yield much more and diverse concrete commonsense knowledge that can serve as an augmentation of the training dataset, while current methods undervalue such a process and only focus on conceptualization. Finally, the complex contextualization and conceptualization of commonsense knowledge can easily bring more than two orders of magnitude of data on top of the original dataset. This makes current labeled data scarce and infeasible for practitioners to annotate all of them, leaving a large amount of unlabeled data.
To fill in these research gaps, we propose CAT (Contextualized ConceptuAlization and InsTantiation), a semi-supervised learning framework that unites event conceptualization and instantiation in cascade to conceptualize CSKBs and acquire abstract commonsense knowledge to aid commonsense reasoning. Inspired by how humans learn with concepts (Carey, 2004), we design a
novel bootstrapping1 method to enhance conceptualizations and abstract commonsense knowledge verification with the help of similar conceptualizations and instantiations as a reference. We demonstrate the effectiveness of CAT by using the acquired abstract commonsense knowledge to train COMET (Bosselut et al., 2019), a commonsense inference language model that generates if-then commonsense knowledge, and showing that our derived abstract commonsense knowledge can significantly improve commonsense inference modeling.
Our contributions are three-fold: (1) We introduce a semi-supervised learning framework, CAT, to conceptualize CSKBs with the assistance of progressively bootstrapping similar abstract concepts or instantiations in the conceptualization process. (2) We use CAT to acquire abstract commonsense knowledge at scale with high quality, which can be used for commonsense inference modeling. (3) We demonstrate the effectiveness of our framework by achieving state-of-the-art performance on two CSKB conceptualization tasks and remarkably improving commonsense inference modeling with our derived abstract commonsense knowledge.

heading: 2 Related Works
text: Conceptualization and Instantiation. Many existing works have studied conceptualization and instantiation separately. Durme et al. (2009) first attempted to derive more general knowledge by abstracting over large sets of factoids obtained from WordNet (Miller, 1995) synsets. Song et al. (2011, 2015) and Gong et al. (2016) proposed to turn instances in a sentence into concepts via weight matching from Probase (Wu et al., 2012). Recently, Liu et al. (2022c) proposed a taxonomy-guided induction method to mine verb-oriented commonsense knowledge from verb phrases. Peng et al. (2022) constructed a conceptual knowledge benchmark to evaluate language models with three zeroshot probing tasks. While these works focus on the conceptualization of entities, He et al. (2022) constructed an event conceptualization benchmark based on ATOMIC (Sap et al., 2019a) by combining syntactic parsing, semantically heuristic matching, and human annotation. Besides, the line of works focusing on ultra-fine entity typing (Choi et al., 2018; Dai et al., 2021; Li et al., 2022) shared
1Bootstrapping refers to the linguistics term in language acquisition that humans learn new knowledge by recognizing its semantic elements and connecting them with known knowledge (Pinker and MacWhinney, 1987).
similar objectives of typing named entities, nominal nouns, and pronouns into a set of free-form phrases. Instantiation was attempted by Allaway et al. (2023), who proposed a controllable generative framework to probe valid instantiations for abstract knowledge automatically. Though Porada et al. (2021) and Peng et al. (2022) both proved that existing pretrained language models lack conceptual knowledge, none of existing works explicitly combine both techniques to derive abstract knowledge that is context-sensitive and generalizable.
Commonsense Reasoning. Endowing NLP systems with the ability to perform commonsense reasoning is an elusive goal of artificial intelligence (Sap et al., 2020). A diverse collection of commonsense reasoning tasks have been proposed as evaluation benchmarks (Talmor et al., 2019; Omura et al., 2020; Ponti et al., 2020; Fang et al., 2021a). Among them, Bosselut et al. (2019) proposed a generative model, COMET, to learn to produce if-then commonsense knowledge as an effective approach toward modeling commonsense inference that can be applied in various commonsense reasoning tasks (Talmor et al., 2019).
Semi-Supervised Learning. Semi-supervised learning (SSL) aims at taking advantage of unlabeled data to equip models with stronger generalization ability (van Engelen and Hoos, 2020). The most common approach is using pseudo labels (Iscen et al., 2019; Wang et al., 2022) to expose more unseen data to the student model. It has been applied in various machine learning tasks such as image classification (Liu et al., 2022a; Hu et al., 2021), text classification (Li et al., 2021; Meng et al., 2019; Xiao et al., 2019), commonsense knowledge base population (Fang et al., 2022), and named entity recognition (Liu et al., 2021; Chen et al., 2021).

heading: 3 Problem Definition
text: Definition. Conceptualizing an event-centric CSKB to derive abstract commonsense knowledge
comprises two steps (He et al., 2022): event conceptualization and triple conceptualization.
Denote the triples in the original CSKB as Do = {(ho, r, t)|ho ∈ Ho, r ∈ R, t ∈ T}, where Ho, R, and T are the set of heads, relations, and tails in the original CSKB. The first step only operates on head events without considering the context in r and t. The goal of event conceptualization is to produce conceptualized head event ha from the original head ho to represent an abstraction of ho. In the second step, the task is to verify whether the conceptualized head ha still makes sense in the context of r and t, as r and t will further restrict the level of abstractness in ha. As shown in Figure 1, conceptualizing watch football game to observe is wrong within the context of having feel relaxed as a result. Plausible (ha, r, t) triples will be considered as valid abstract commonsense knowledge.
Specifically, in the first step, there are two ways of conceptualizing head events alone: a retrievalbased discriminative way and a generative way. The retrieval-based discriminative paradigm identifies and links a component i in ho to a concept c in a concept taxonomy C to form a conceptualization ha by replacing i with c. The model needs to verify whether ha is a valid conceptualization of ho. The generative paradigm aims to generate a ha directly given ho and the designated component i in ho.
Formally, denote the annotated dataset in the first step, event conceptualization, as Dlh = {(ho, ha, y)|ho ∈ Ho, ha ∈ Ha, y ∈ {0, 1}}, where ho is an original head event without conceptualization, ha is a corresponding conceptualization of ho, and y is the human-annotated label indicating whether such a conceptualization is plausible or not. The labeled dataset in the second step, triple conceptualization, is denoted as Dlt = {(h, r, t, y)|h ∈ Ha, r ∈ R, t ∈ T, y ∈ {0, 1}}, where h is a conceptualized head event from the first step, r and t are a relation and a tail from the original CSKB accompanied with the corresponding original head ho, and y is the human-annotated label indicating whether such abstract commonsense knowledge, in the form of a conceptualized triple, is plausible or not. Besides labeled datasets, unlabeled datasets are defined similarly as Duh and Dut only with the difference that labels y are missing. Thus, the task objective for discriminative event conceptualization is to determine whether a ho can be properly abstracted using ha, where ha is derived by replacing a component i ⊂ ho with
its linked concept c from a concept taxonomy C. The task objective for generative event conceptualization is to generate ha directly from ho with text generation models. For the triple conceptualization task, the objective is to distinguish whether a conceptualized triple (ha, r, t), representing abstract commonsense knowledge, is plausible or not.
Dataset. To study conceptualization over CSKBs, we use the AbstractATOMIC dataset provided by He et al. (2022) as the benchmark. In AbstractATOMIC, ATOMIC is used as the original CSKB. And the event conceptualization adopts a discriminative way, where a syntactic parsing schema is defined to identify the components i in ho to be heuristically linked to concept taxonomies Probase (Wu et al., 2012) and WordNet (Miller, 1995) to form conceptualized ha. Such a heuristic can produce over 32 times more candidate conceptualized head events and over 10 times more conceptualized triples compared with the original ATOMIC, as the number of retrieved concepts from the concept taxonomy C can be manually controlled to acquire a large number of conceptualizations. Triple conceptualization is defined as predicting the plausibility of the triples whose head is conceptualized. Only 131K (26%) conceptualizations of 7K (45%) ATOMIC head events and 81K (1.3%) conceptualized triples are manually anno-
tated as Dlh and D l t, while others remain unlabeled Duh and D u t . The trn/dev/tst partition follows the same split as in the original ATOMIC. Statistics and more detailed explanations of AbstractATOMIC are shown in Table 1 and Appendix A.

heading: 4 CAT Framework
text: This section introduces our proposed Contextualized ConceptualizAtion and InsTantiation (CAT) framework for conceptualizing commonsense knowledge bases and acquiring abstract commonsense knowledge. An overview is presented in Figure 2. Our motivation is two-fold: first, adding instantiation after conceptualization to form a cycle can strongly benefit two conceptualization tasks simultaneously. On the one hand, instantiating conceptualized triple relies on the correctness of event conceptualization. On the other hand, properly conceptualized triples can benefit event conceptualization via instantiation by providing more context brought by (r, t). Second, to address the lack of annotations, we resort to pseudo labeling, a typical semi-supervised learning approach to automatically assign pseudo labels to the vast majority of unlabeled data using a teacher model.
Following He et al. (2022), we study the retrieval-based discriminative paradigm of event conceptualization and leave the generative paradigm as an intrinsic evaluation. In CAT,
we unify event conceptualization and triple conceptualization into one cycle and make them mutually benefit each other through instantiation and conceptualization. Our framework can be summarized into four steps: (1) Train teacher models for both event conceptualization and triple conceptualization on the labeled dataset Dlh and D l t, respectively. Use the two teachers to assign pseudo labels to unlabeled datasets. (2) Conduct alternative conceptualization or instantiation on labeled and pseudo-labeled data. (3) Bootstrap (aggregate) the alternative concepts and instances in the second step using natural language prompt templates and train student models on both labeled and pseudo-labeled data. (4) Use the student models to refine the pseudo labels and then re-train the student models.

heading: 4.1 Teacher Model Training
text: Two teacher models on both event and triple conceptualization tasks are trained separately on the labeled dataset Dlh and D l t. As both tasks are inherently text/triple classification, we adopt KGBERT (Yao et al., 2019) as the skeleton of our models. The event conceptualization model determines whether ha is a valid conceptualization of ho, and the triple conceptualization model determines whether a conceptualized triple (ha, r, t) is plausible or not. The two models θ are trained on annotated examples xi with a cross-entropy loss (Eq. 1) and used to provide pseudo labels to instances from the unlabeled datasets Duh and D u t . Two thresholds, T+ and T−, are set to determine the pseudo labels of unlabeled examples with high confidence. Examples with a pseudo-labeled score higher than T+ will be labeled yi = 1, and those lower than T− will be labeled yi = 0. The rest will be discarded.
L(xi, θ) = − |x|∑
i=1
yi log(θ(xi)) (1)

heading: 4.2 Alternative Conceptualization and Instantiation
text: According to Murphy (2004), when humans learn a new concept, we pre-extract similar known concepts in our minds and infer possibly equivalent unknown concepts on the fly. Inspired by this theory, we retrieve additional abstract concepts or instantiated events to help discriminate conceptualizations and abstract commonsense knowledge. For event conceptualization, we retrieve some alternative pos-
sible conceptualizations of ho to accompany the learning of ha. Additional conceptualizations of ho from both labeled and pseudo-labeled examples are predicted again by the teacher model and ranked according to their plausibility score prediction. And top m conceptualizations are retrieved with m being a hyperparameter to control the number of retrievals. For triple conceptualization, we perform instantiation in cascade to instantiate c to some concrete instances to assist the learning process. Possible instantiations of c are extracted from annotated and pseudo-labeled event conceptualizations by searching for conceptualized events h′a ∈ Ha other than ha with c as the concept and extracting their corresponding instances i ⊂ h′a. Similarly, the instances are then scored by the teacher model, and the top n of them are retrieved. Intuitively, alternative event conceptualizations can serve as hints for discriminating the correctness of the target conceptualization, and instantiations can carry additional contextualized information to help verify the plausibility of a conceptualized triple, which meets the objective of deriving abstract commonsense knowledge that is context-sensitive.

heading: 4.3 Prompt Aggregation
text: We then bootstrap the retrieved alternative conceptualizations/instantiations via natural language prompts. Here bootstrap (Carey, 2004) can be understood as binding the alternative retrievals and the target concept/triple together to strengthen the discrimination of the target concept/triple. As shown in Figure 2 step (3), the initially given input and retrieved concepts/instances are concatenated via human-defined prompts for both conceptualization tasks. Alternative concepts/instances are sorted in the order of their plausibility score ranking. Two student models Sh and St for both tasks are trained using the modified text with such prompts as inputs. They are expected to learn the bootstrapping connectionism between the target and the additional retrievals we provided. More detail about the prompt design is in Appendix B.

heading: 4.4 Pseudo-Label Refinement
text: All pseudo labels, initially derived by a teacher model trained on the original labeled dataset, are relabeled according to the plausibility score predicted by our newly enhanced student models Sh and St. Similar to the teacher model, two thresholds, T+ and T−, are applied to distinguish positive and negative examples for both tasks. In addition, negative
labels are assigned to triples whose conceptualized head events are predicted as wrong conceptualizations by Sh, as wrong conceptualizations will not yield plausible abstract commonsense knowledge.

heading: 4.5 Application and Evaluation of CAT
text: The resulting models of CAT include an event conceptualization model and a triple conceptualization model, both fine-tuned on the refined pseudo labels and the labeled data. These two models can be used to conceptualize ATOMIC to a larger commonsense knowledge base on a more abstract level. We further conduct intrinsic evaluations on the acquired event conceptualization model under a generative event conceptualization paradigm and extrinsic evaluations on the resulting conceptualized CSKB with commonsense inference modeling task (COMET; Bosselut et al. (2019)) in Section 5. Here we select COMET as the representative because it is a general commonsense model that can be applied to various downstream commonsense reason-
ing tasks such as SocialIQA (Sap et al., 2019b), self-talk (Shwartz et al., 2020), and CSKB completion (Malaviya et al., 2020). Meanwhile, generative event conceptualization enables performing automatic conceptualization scalably. Both are important applications and evaluations of CAT.

heading: 5 Experiments
text: We conduct conceptualization experiments using CAT in Section 5.1 and generative experiments as evaluations in Section 5.2. These experiments demonstrate that CAT has a strong capability in conceptualizing CSKBs, and better conceptualization modeling can help populate more novel and diverse commonsense knowledge and thus help commonsense modeling (COMET).

heading: 5.1 CSKB Conceptualization
text: Baselines. We collectively introduce the baselines for both event and triple conceptualization tasks, as they are inherently classification tasks.
AUC is used as the evaluation metric. Under a supervised learning setting, we apply KG-BERT (Yao et al., 2019) model with BERT (Devlin et al., 2019), BART (Lewis et al., 2020), RoBERTa (Liu et al., 2019), DeBERTa (He et al., 2021, 2023), and ELECTRA (Clark et al., 2020) as the backbone language models. We also attempt to leverage supervised generative language models as baselines. GPT2 (Radford et al., 2019) models are trained with a text generation objective only on positive examples, and we use perplexity as the prediction scores to calculate AUC. For the semi-supervised learning baselines, we leverage UDA (Xie et al., 2020a), NoisyStudent (Xie et al., 2020b), and PseudoReasoner (Fang et al., 2022) with RoBERTalarge being the backbone model. Additional explanations can be found in Appendix C.1.1.
Discriminative Results. The results for both tasks are presented in Table 2. Under a supervised learning setting, KG-BERT family mostly performs better on both tasks than GPT2 due to the fact that GPT2 is only fine-tuned on positive examples and thus cannot learn from negative examples that contain wrong conceptualizations and implausible abstract commonsense knowledge. As for the semi-supervised learning setting, previous SSL baselines are rather limited in improving the performance against supervised learning. The best PseudoReasoner only improves by 0.5% and 0.3% on the test set for both tasks compared with supervised RoBERTa-large models. Instead, models trained with CAT can outperform all other training methodologies. Comparing the test set performance with PseudoReasoner, small backbone models (BERTbase) can improve by 3.4% and 2.2%, and large models (RoBERTa-large) can be improved by 2.1% and 2.2%. This shows pipelining two-step concep-
tualizations as a loop and leveraging our proposed bootstrapping-based method can yield a larger performance gain compared with simply applying a semi-supervised learning strategy. Due to limited space, ablation studies on framework components and the semi-supervised learning paradigm of CAT are conducted in Appendix C.1.4. For example, the results indicate that bootstrapping alternative conceptualization and instantiation plays the most important role in assisting learning conceptualization among all components of CAT. Additional results and a computational cost study can be found in Appendix C.1.3 and Appendix D.

heading: 5.2 Application and Evaluation of CAT
text: As CAT is a framework for acquiring conceptualized commonsense knowledge, including both conceptualized head events (from ho to ha) and abstract commonsense triples (ha, r, t), we assess these pseudo-labeled outcomes via two generative tasks with various threshold tuning as evaluations.
Generative Event Conceptualization. To intrinsically evaluate the effectiveness of CAT’s event conceptualization, we use the acquired conceptualized head events as training data to learn a generative event conceptualizer. Specifically, the models are trained with instance-conceptualizations pairs in the format of “<instance> is an instance of <concept>”. At the evaluation phase, the model is prompted with “<instance> is an instance of [GEN]” where <instance> is the instance to be conceptualized and [GEN] is the generation token. We then retrieve the top-1 generation and compare it against the target set from the evaluation dataset to compute four NLG metrics, as listed in Appendix C.2.1. These scores can be regarded as an approximation of the top-1 generations’ recall.
Additionally, we uniformly sample 500 generations from each evaluation split and conduct expert annotations on the plausibility of each conceptualization to ensure that out-of-domain concepts can be properly evaluated. The experts are asked to determine whether each top-1 generation is indeed a plausible conceptualization or not, such that the top-1 generations’ precision is reflected. Thus, current evaluation measures jointly evaluate the top-1 generations’ precision and recall, which makes it robust and non-easy to be impacted by repetition problems (Li et al., 2020). Zero-shot GPT2 and GPT2 fine-tuned on the originally labeled event conceptualizations in Dlh are used as baselines. We also study the effect of the threshold T+ that selects plausible conceptualized heads, where higher thresholds indicate higher plausibility regarded by CAT. The results are presented in Table 3. With a relatively high threshold, generators trained on a mixture of pseudo-labeled data by CAT and annotated concepts significantly outperform the baselines in every automated metric. A plausible rate of 93.3% is maximally achieved on the test set, which is 11.8% higher than the baseline. Gradually reducing the threshold also decreases the performance, indicating abstract heads with lower plausibility scores can be of poorer quality. Such results indicate that CAT can produce high-quality event conceptualizations for generative models to learn better conceptualizers without the need to annotate a large number of data.
Commonsense Inference Modeling (COMET). The second component of CAT produces triple-
level abstract commonsense knowledge. We evaluate these abstract commonsense triples with a commonsense inference task that generates commonsense tails given heads and relations as inputs, as in COMET (Bosselut et al., 2019). Following He et al. (2022), we apply the same training and evaluation process to the models. The base training data we use are a subset of ATOMIC triples corresponding to those annotated abstract triples in Dlt, which contains 17K (3.7%) among the original ATOMIC. We derive abstract commonsense knowledge using CAT from a subset of Dut where the heads correspond to those in the ATOMIC subset to ensure no data leakage, denoted as DuCAT. GPT2 is fine-tuned on the ATOMIC subset, the annotated abstract triples Dlt, the abstract knowledge verified by CAT, or their combinations. The commonsense generation results are presented in Table 4. Similar to COMET (Bosselut et al., 2019), all models are evaluated on the original ATOMIC’s full validation and testing sets. The best result is achieved using a mixture of the ATOMIC subset and abstract triples pseudo-labeled by our framework, with 0.95 as the threshold for selecting plausible triples. This indicates high-quality abstract commonsense triples can indeed provide a more general view of the original commonsense knowledge, thus helping commonsense inference. Additionally, training with our pseudo-labeled examples outperforms training with those annotated triples in AbstractATOMIC, which also validates the effectiveness of our model that leverages a large amount of unlabeled data. To further investigate how conceptual knowledge
improves commonsense inference modeling, we conduct more empirical analysis in Section 5.4. Additional experiment results with other thresholds and case studies can be found in Appendix C.2.3 and Appendix E, respectively.

heading: 5.3 Number of Retrieved Alternative Conceptualizations and Instantiations.
text: We then study the ablation of bootstrapping different numbers of alternative conceptualizations/instantiations (denoted as #retrieval) in our CAT framework. For simplicity, when tuning the #retrieval for one task, the #retrieval of the other task is fixed at the best value we acquired. We plot the test AUC score with #retrieval from 0 to 11 using BERT-base as the backbone model in Figure 3. #retrieval=0 refers to training with a simple student-teacher framework without bootstrapping alternative conceptualizations and instantiations. For event conceptualization, the performance generally positively correlates with the number of retrievals, while it starts dropping after 9. A reversed trend is observed for triple conceptualization, where using only two instances achieves the best performance. One possible reason is that in triple conceptualization, the retrieved instances are events and much longer than the retrieved concepts in event conceptualization, and aggregating various alternative events for a triple will cause language models to be less sensitive to the semantics of the original triple (Holtzman et al., 2020).

heading: 5.4 The Effect of Abstract Knowledge
text: We finally study the effect of abstract commonsense knowledge acquired by CAT by studying the semantic overlaps between training and testing data. We sort the test set by the BERTScore (Zhang
et al., 2020b) between each individual testing entry against the whole training set in the original ATOMIC and split them in half to acquire two test groups. The testing entries with lower BERTScore on the training set indicate a larger semantic shift from the training set (Deutsch and Roth, 2021), which is also harder for models to discriminate (Hsu et al., 2020). We denote the testing group with a lower BERTScore as “Difficult” and the other half as “Easy”. The performance gain on the two test set splits between the best conceptualization-aided COMET and the COMET trained on the ATOMIC subset only is reported in Figure 4. We can observe that training COMET with abstract commonsense knowledge leads to a larger improvement for harder test examples dissimilar from the original training set, indicating that introducing extra abstract commonsense knowledge can help COMET become more generalizable to harder test sets.

heading: 6 Conclusion
text: In conclusion, this paper proposes CAT, a semisupervised learning framework for commonsense reasoning, by leveraging the power of abstract commonsense knowledge. By achieving state-of-theart performances in CSKB conceptualization tasks, we remarkably improve modeling commonsense inference, as an important cornerstone of many commonsense reasoning tasks. Our analysis also demonstrates that high-quality abstract commonsense knowledge can benefit commonsense inference modeling by providing more generalizability on hard commonsense knowledge. We hope this work can draw insights toward commonsense reasoning from a conceptualization perspective.
Limitations
Our framework manually sets thresholds T+ and T− in pseudo labeling by observations of data quality and hyperparameter searching. Dynamic threshold tuning (Xu et al., 2021) or meta pseudo labels (Pham et al., 2021; Li et al., 2021) can be implemented to better filter pseudo-labeled examples. And the thresholds for different tasks can be tuned separately to improve the models’ generalizability.
Recently, large generative language models such as GPT3.5 (Brown et al., 2020) and ChatGPT2 (Ouyang et al., 2022; Gao et al., 2022) have demonstrated their strong potential on various NLP tasks including probing abstract commonsense knowledge with in-context learning (Brown et al., 2020; Xie et al., 2022). Due to our limited access, we did not conduct fully-scaled experiments in our paper. A short discussion with case studies is provided in Appendix E.3.
While our framework only operates on AbstractATOMIC as the conceptualization of ATOMIC, it’s also worthy of verifying our framework on other CSKBs such as ATOMIC2020 (Hwang et al., 2021), GLUCOSE (Mostafazadeh et al., 2020), ATOMIC10X (West et al., 2022), FolkScope (Yu et al., 2022a) and eventuality CSKB such as ASER (Zhang et al., 2020a, 2022) and constructing large conceptualized CSKB benchmarks. In addition, we only evaluated the power of the acquired abstract commonsense knowledge on the commonsense knowledge generation task (COMET), while other commonsense reasoning tasks remain future works (Wang et al., 2023a), such as COLA (Wang et al., 2023b), CommonsenseQA (Talmor et al., 2019, 2021), SocialIQA (Sap et al., 2019b), Winograd Schema Challenge (Levesque et al., 2012), PIQA (Bisk et al., 2020), Abductive Commonsense Reasoning (Bhagavatula et al., 2020), and Winogrande (Sakaguchi et al., 2020).
Ethics Statement
This paper introduces CAT, a framework for commonsense reasoning via conceptualizing CSKB to acquire abstract commonsense knowledge. The experiments are conducted on publicly available and well-established datasets that are shared via open-access licenses. The usage of these datasets in our paper is only for research purposes and is consistent with the datasets’ intended usage. The
2https://chat.openai.com/
primary dataset, AbstractATOMIC, largely shares the content with another CSKB, ATOMIC, which is anonymized and desensitized (Sap et al., 2019a). Thus, no data privacy issue is involved.
The potential risks of CAT are relatively low. Since CAT is trained on AbstractATOMIC, a conceptualization benchmark based on a popular CSKB, ATOMIC, and two concept taxonomies, Proabse and WordNet, it is expected that CAT does not contain any private, offensive, biased, and sensitive information or social, political issues. The studied tasks all focus on conceptualization or CSKB, which is not likely to generate harmful content, as shown in the case studies in Appendix E. Thus, we believe that CAT does not yield additional risks.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers for their constructive comments. The authors of this paper are supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20), and the GRF (16211520 and 16205322) from RGC of Hong Kong, the MHKJFS (MHP/001/19) from ITC of Hong Kong and the National Key R&D Program of China (2019YFE0198200) with special thanks to HKMAAC and CUSBLT. We also thank the UGC Research Matching Grants (RMGS20EG01-D, RMGS20CR11, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08).

heading: A Dataset Description
text: In this section, we introduce more about AbstractATOMIC (He et al., 2022), as the primary dataset we experimented with. AbstractATOMIC is a conceptualized commonsense knowledge benchmark that is built upon ATOMIC (Sap et al., 2019a), a popular CSKB in the format of (h, r, t) triples. The dataset is entirely in English. It contains two parts of data: (1) event conceptualization data and (2) abstract knowledge triples conceptualization data.
The event conceptualization data contain conceptualizations for head event instances, where the events are filtered from the original ATOMIC head events. Unlike the traditional entity concept taxonomies, where instances are nouns or verb phrases, AbstractATOMIC includes instance candidates that can be either the entire head event or a certain component of an event. Detailed examples can be found in Appendix E.
The instances within each head event are identified through syntactic parsing by using a parser from the spaCy 3 library and matching with five human-defined rules. After identification, the candidate instances will be heuristically matched against Probase (Wu et al., 2012) and WordNet (Miller, 1995) via GlossBERT (Huang et al., 2019) to acquire their candidate concepts. A neural generator based on GPT2, similar to the baseline in this paper, is also trained to generate concepts. A supervised conceptualization verifier, based on RoBERTa (Liu et al., 2019), is trained as the final gatekeeper to verify the acquired concepts roughly.
Human annotations on the Amazon Mechanical Turk platform are further conducted to acquire annotations on the correctness of 131K conceptu-
3https://spacy.io/
alizations of 7K ATOMIC events. All conceptualizations that are not annotated are regarded as unlabeled data in this paper. More detailed statistics for the head event conceptualization data can be found in Table 5.
After acquiring the event conceptualizations by only focusing on head events, abstract commonsense knowledge, in the form of (h, r, t) triple, is collected by connecting conceptualized head event with its non-abstract counterparts (commonsense relations and inference tails) from ATOMIC. Only the head events contain abstract concepts. Thus, these abstract triples are more generalized if-then commonsense knowledge that is potentially useful for commonsense reasoning through instantiation.
Human annotations on Amazon Mechanical Turk further verify 81K uniformly sampled abstract triples. These triples only correspond to 689 unique ATOMIC head events, which makes annotations relatively scarce compared with the scale of unlabeled data. A supervised RoBERTalarge verifier is trained on the annotated triples to roughly verify abstract triples that are not annotated. Triples with scores higher than 0.9 are pseudo-labeled as positive ones (He et al., 2022). However, this paper only leverages these pseudolabeled examples in the commonsense inference generation task (COMET) as baselines. Only annotated triples are considered hard-labeled for all other tasks concerned. And triples that are not annotated are treated as unlabeled by default. The detailed relational distribution of abstract triples is presented in Table 6. Examples can be found in Appendix E.

heading: B Prompt Design
text: In this section, we introduce the textual prompts used for training various models.
For event conceptualization, denotes the original event as ho, instance as i, target concept to be verified as c, and retrieved alternative conceptualizations as cr,1, cr,2, cr,3, ..., cr,m. The prompt for training the teacher model is “[CLS] ho [SEP] c”, while the one for training the student model is “[CLS] ho [SEP] c [SEP] cr,1, cr,2, cr,3, ..., cr,m”. For the example in Figure 2, the filled prompt is “PersonX is on vacation [SEP] relaxing event [SEP] traveling, break, holiday.” Specifically, special tokens <c> and </c> are used to enclose i ⊂ ho within the original event to highlight the instance to be conceptualized. GPT2 generators use similar prompts, with the difference that [SOS] and [EOS] special tokens are inserted to denote the start and end of the sentence, respectively.
For triple conceptualization, denotes the head, relation, and tail of an abstract commonsense triple as (h, r, t), the abstract concept in the conceptualized head as c ⊂ h, and retrieved instantiations as er,1, er,2, er,3, ..., er,n. The prompt for training generally follows the one used by He et al. (2022). For the teacher model, “[CLS], h1, ..., h|h|, [SEP], [r], [SEP], t1, ..., t|t|” is used as the prompt. Similarly, student models are trained with a prompt “[CLS], h1, ..., h|h| [SEP] [r] [SEP] t1, ..., t|t| [SEP] er,1, er,2, er,3, ..., er,n”. A filled example by using the case in Figure 2 is “relaxing event [SEP] because PersonX wanted [SEP] have fun [SEP] PersonX joins party, go on a holiday, Take a break.” The commonsense relation within each triple is translated into human-readable text, as shown in Table 7.
The generative event conceptualization by GPT2 generators uses “[SOS] ho [SEP] i [GEN]” as the input template, where [GEN] indicates the special
token for generation. Commonsense inference modeling uses the same prompt as done by Hwang et al. (2021); Fang et al. (2021b).
In addition, we observe that adding special tokens such as <c> and </c> can effectively boost performance. But adding textual guidelines such as “is an instance of” or “is a concept of” does not have any positive effect. The same trend is observed for the bootstrapping prompt, where adding external texts such as “is also instances of” or “can be instantiated to” will harm the model significantly.

heading: C Additional Experiments
text: In this section, we present additional details and experiment results for CSKB conceptualization tasks (Appendix C.1) and applications, as well as evaluations, of CAT (Appendix C.2) that are not covered in the paper due to limited space.
C.1 CSKB Conceptualization C.1.1 Baselines For supervised learning baselines of both discriminative conceptualization tasks, KG-BERT (Yao et al., 2019) is adapted as the skeleton of our baseline models. For BART, we use the embedding of the end-of-sentence token in the decoder as the representation of the input sequence. For other models, the embedding of the [CLS] token is used as the representation vector. Linear layers are appropriately appended after the encoder model to perform text classification.
For the semi-supervised baselines, we provide additional explanations for different methods:
UDA. In the original paper of UDA (Xie et al., 2020a), two data augmentation methods, backtranslation and TF-IDF replacement, are implemented for unsupervised data augmentation. We leverage both methods in our conceptualization tasks as two different baselines. For the triple conceptualization task, we follow the same setting as proposed in PseudoReasoner (Fang et al., 2022). The back-translation method translates the original corpus from English to French and then translates it back. Special replacements are taken to avoid the influence of special tokens. Meanwhile, the TF-IDF method uses a probability of 0.1 to replace the original corpus according to its TF-IDF score. For the event conceptualization task, we concatenate the head event and its annotated concept into one new sentence and then feed it into the model. For the unlabeled conceptualizations, we enclose
the instance and concept with special tokens <c> and </c>, which is the same as our framework, and then use back translation or TF-IDF to generate the augmented data. The input for triple conceptualization follows a similar way as supervised baselines. It is observed that these special tokens will not affect the translation significantly as they will be preserved in the translation output. Last but not least, the model θ is trained on a mixture of annotated data x1 and augmented data x2 by using the consistency training loss, as shown in Equation 2.
J(θ) = Ex1∼PL(x)[− log pθ(y1|x1)]+ λEx2∼PU (x)Ex̂∼q(x̂|x2)[CE(pθ̃(y|x2)||pθ(y|x̂)]
(2)
NoisyStudent. Noisy Student (Xie et al., 2020b) is an iterative training method that leverages a teacher-student paradigm. The teacher model is first trained on annotated data. It is then asked to make predictions on the unlabeled data as pseudolabels. Then, another student model with an equal or larger number of parameters is trained with a mixture of annotated and pseudo-labeled data. Note that pseudo labels, in numerical values, are directly used as the targeting labels. The trained student model will serve as a new teacher and re-label the unlabeled data again to yield a better prediction. In our implementation, dropout or dynamic model depth is introduced as noise to the model. All models θ are trained with standard cross-entropy loss, as shown in Equation 1. We set the dropout probability to 0.5, as it leads to the fastest convergence on our data. Only one iteration is completed in our experiment, as that’s when the student model reaches its best result.
PseudoReasoner. PseudoReasoner (Fang et al., 2022) is another iterative semi-supervised learning framework that is proposed to tackle Commonsense Knowledge Base Population (CKBP) task (Fang et al., 2021a, 2023). It leverages a similar teacher-student paradigm and a novel filtering mechanism with the assistance of the student model. We replaced the generative teacher model with a DeBERTa-v3-large model due to the disastrous performance that GPT2 achieved on both verification tasks. Similar to CAT, two thresholds, T+ = 0.9 and T− = 0.1, are determined to assign pseudo-labels to unlabeled data based on the prediction of the teacher model. The rest steps remain the same as described in the original paper. Similar to
NoisyStudent, only one iteration is carried out for PseudoReasoner as the student model converges to the best.
C.1.2 Settings We use pretrained language models from the Huggingface Transformers4 Library (Wolf et al., 2020) to build our framework. The learning rate for all models is set as 5e-6, and the batch size is 64. We use an AdamW (Loshchilov and Hutter, 2019) optimizer and evaluate the model every 25 steps. The max sequence length for the tokenizer is set to 25 and 35 for both discriminative tasks, respectively. Due to the imbalanced dataset, we evaluate the discriminative models with Area Under Curve (AUC) score (Bradley, 1997). Early stopping is used where the best checkpoint is selected when the largest validation AUC is achieved. All experiments are repeated three times using different random seeds, and the average performances and standard deviations are reported. In addition, we set the probability thresholds for both tasks to T+ = 0.9 and T− = 0.1 to determine the pseudo labels. The thresholds are roughly derived by observing the overall distribution and quality of data satisfying the respective threshold. For the bootstrapping method, we bootstrap m = 9 additional concepts for event conceptualization verification and n = 2 additional instances for abstract triple verification. Detailed ablation studies are provided in Section 5.3. As for the computational infrastructure, the models are trained and evaluated on four NVIDIA RTX3090 (24G) and four NVIDIA 1080Ti (12G) graphical cards. The number of parameters for every model is reported in Table 11.
C.1.3 Additional Experiment Results The full experiment results for discriminative CSKB conceptualization tasks are reported in Table 11. All supervised learning baselines achieve comparable results as reported by He et al. (2022). Supervised CAT will be discussed later. The results by semi-supervised CAT are generally consistent with our findings as discussed in Section 5.1. To study the effect of different components and the training regime of CAT, we conduct more detailed ablation studies in Appendix C.1.4.
C.1.4 Ablation Study In this section, we study the effects of different components in CAT and the training strategy of
4https://huggingface.co/docs/transformers
CAT. These studies indicate that our framework design and the proposed bootstrapping method play an important role in CSKB conceptualization and are more effective than leveraging unlabeled data with pseudo labels.
Framework Components. Our CAT framework consists of three critical components that make CAT different from traditional semi-supervised baselines. They are denoted as: • Bootstrapping: Assist the training of student models by retrieving alternative conceptualizations and instantiations and bootstrapping them via natural language prompts. Dropping this component will train student models with the original textual prompts that are also used by the teacher models. • CAT Cycle: Unite event and triple conceptualization tasks by assigning negative pseudo labels to abstract triples whose conceptualized head is predicted as wrong conceptualization. Dropping this component will separate the framework into two lines of training, which are training event conceptualization and triple conceptualization models separately. • Pseudo-label refinement: Refine the pseudo labels with the latest student models and re-train the student models. Dropping this component will not update any pseudo label and will not re-train the student model.
We then conduct ablation studies regarding these three components with semi-supervised CAT to prove the effectiveness of our framework design and proposed bootstrapping method. Each component is removed separately, and the test set performances by student models are reported. The results are shown in Table 8. From the results, bootstrap-
ping alternative conceptualization and instantiation leads to the largest performance gain. Bridging event conceptualization discrimination with triple conceptualization also causes slight improvements. However, refining the pseudo labels and re-train the student models have barely any effect. Thus, our bootstrapping method is the most important component within the entire CAT framework and can effectively assist in learning conceptual knowledge.
Supervised CAT. We further study training CAT in a supervised learning setting to examine the role of unlabeled data. In supervised CAT, no teacher models are trained to provide pseudo labels. The alternative conceptualizations and instantiations are retrieved directly from the annotated event conceptualization data and bootstrapped later. Two student models are trained on the bootstrapped data only and evaluated on the same testing set, and the results are reported in Table 11. Compared with supervised learning baselines, supervised CAT can achieve a comparable result on the event conceptualization task. This may be due to the fact that the diversity of concepts drops without considering unlabeled conceptualizations. Improvements in the triple conceptualization task are more significant, and the results are comparable with semisupervised CAT. This indicates that our framework design and bootstrapping method are successful in discriminating high-quality abstract commonsense knowledge, and leveraging a semi-supervised learning paradigm benefits more in event conceptualization discrimination.
C.2 Application and Evaluation of CAT
C.2.1 Settings Pretrained GPT2 models from the Huggingface Transformers Library and training codes5 by Hwang et al. (2021) are used as our code base. The learning rate for all experiments is set to 1e5, and the batch size is fixed to 64. We use an Adam (Kingma and Ba, 2015) optimizer and evaluate the model every 20 steps. The input and output lengths for GPT2 models are fixed at 45 and 55 for the two application and evaluation tasks, respectively. Such length settings can cover all annotated conceptualizations and triples. For both generative experiments, we evaluate the generations with BLEU (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), ROUGE-L (Lin, 2004), and
5https://github.com/allenai/comet-atomic-2020
CIDEr (Vedantam et al., 2015) scores. However, since an abstract concept usually contains one or two tokens, we only report BLEU1 and BLEU2 scores for the generative event conceptualization task. Early stopping is also applied where the best checkpoint is selected when the minimum autoregressive LM loss is achieved. In addition, we notice that the number of triples from the ATOMIC subset is much smaller than abstract triples for the commonsense inference modeling task. Thus, we upsample the ATOMIC subset by a ratio of 1:2 across all experiments to guarantee a consistent and balanced number of training data. For generative event conceptualization, the training data is simply a mixture of annotated and pseudo-labeled event conceptualizations without any balancing measure. All the models are trained and evaluated on four NVIDIA RTX A6000 graphical cards with 48G memory. The number of parameters is close to the number of parameters in GPT2-XL, which is reported in Table 11.
C.2.2 Annotation Settings When evaluating the event conceptualization generator, expert annotations are conducted to evaluate concepts that are not presented in the training set. Crowdsourced platforms such as Amazon Mechanical Turk are not used since experts understand conceptualization better and are more reliable for evaluation. Subsequently, the authors of this paper are invited to serve as expert annotators. They are experienced in NLP research and clearly understand the paper’s scope. The annotation guideline is carefully designed. Each question presents the original head event with the instance highlighted and the corresponding conceptualization candidate to be annotated. There are also several positive and negative conceptualizations attached as examples. The authors are well-informed about the instruction and the intended use of their annotations in this paper. And they all agreed to annotate as part of their contributions. Moreover, in order to ensure that the expert will not deliberately raise the plausible rate of a certain set of annotation candidates, we randomly shuffle all the data and invite one more expert to cross-validate the annotations. These measures can ensure that the annotation process is free of ethical concerns and justifiable.
C.2.3 Additional Experiment Results We conduct a more comprehensive study on the commonsense inference generation task by experi-
menting with the effect of threshold tuning when filtering abstract commonsense knowledge. Multiple thresholds ranging from 0.5 to 0.995 are experimented with to derive abstract commonsense knowledge of different qualities. COMET (GPT2XL) generators are fine-tuned on the ATOMIC subset, augmented by a mixture of annotated and pseudo-labeled abstract triples. The performance curve according to the threshold is plotted in Figure 5. Full version results with all metrics are reported in Table 19. It can be observed that gradually increasing the threshold from 0.75 will lead to better performance, which may be due to the improvement in data quality. However, increasing the threshold over 0.95 will cause a performance drop. One possible reason is the amount of pseudolabeled triples significantly drops with a relatively high threshold, and COMET fails to learn well from annotated triples only. Using the CAT framework to pseudo-label unlabeled abstract triples leads to better performance than leveraging a RoBERTalarge supervised discriminator to assign pseudolabels, which also validates the reliability of the triple conceptualization discriminator in CAT. Also, it is noticeable that training COMET with triples based on our constructed ATOMIC subset is much worse than training with the full ATOMIC dataset. This indicates that exposing the model with substantial factual commonsense knowledge is still important, and only equipping the model with abstract commonsense knowledge is not enough for commonsense inference modeling.

heading: D Computational Cost Analysis
text: In this section, we compare the number of training data used for both CSKB conceptualization tasks to compare the computational cost across different frameworks and methodologies empirically. Both annotated and pseudo-labeled data are counted. The comparison result is presented in Table 9. All semi-supervised learning methods leverage a significant amount of unlabeled data due to the great scarcity of annotations. With threshold filterings, PseudoReasoner (Fang et al., 2022) and our CAT framework can abandon more than half of pseudo examples with poor quality. Even though our CAT framework can still outperform PseudoReasoner and achieve the best performance among all methods. Additionally, there is no notable increase in the number of model parameters as CAT also applies a teacher-student paradigm that is similar to Noisy-Student and PseudoReasoner. Even compared with the supervised baselines, CAT only doubles the parameters used. In conclusion, with comparable training data and parameters against other baselines, CAT can achieve much better results and state-of-the-art performances.

heading: E Case Studies
text: This section contains case studies of the four tasks we studied in this paper, including CSKB conceptualization tasks and applications of CAT. Throughout these cases, we would like to offer a clearer view of the data, discuss the challenges of the conceptualization task, and provide brief error analyses.
E.1 CSKB Conceptualization
Event Conceptualization. For discriminative event conceptualization, the case study is shown in Table 15. From these cases, it can be observed that several instances i can be identified within one
head event ho, and each of them can be conceptualized in multiple ways. Formally, assume we are conceptualizing m events, each with n instances. And each instance i concerned can be conceptualized as p concepts. Each concept takes the majority vote of q annotators to verify. Subsequently, the number of annotations needed is O(mnpq), which grows significantly if we conceptualize a commonsense knowledge base at scale. Thus, it is extremely infeasible for practitioners to annotate all of the conceptualizations for verification, which also highlights the importance of a reliable discriminative conceptualization model as CAT acquired. Semi-supervised learning is also an ideal training strategy, as there is a considerable amount of unlabeled data.
Analyzing the errors made by our discriminator, we observe that models frequently make errors when the instance contains the word “PersonX,” which could be caused by the reporting bias (Gordon and Durme, 2013), as “PersonX” is seldom used in normal natural language texts. Replacing the subjects with commonly used names such as “Alex, Bob” may alleviate such a problem. Additionally, models make errors on some rarely seen concepts, such as “organ,” “cognitive ability,” and “side effect.” Their absence from training data can partially cause this, as CSKB, like ATOMIC, may not cover many instances under those rarely used concepts.
Triple Conceptualization. For triple conceptualization discrimination, case studies are shown in Table 17. Similar to the analysis above, consider m events with n instances, each instance with p concepts. Assume that every ATOMIC head event has t relation and tail tuples as its counterpart, and q votes are required from annotators. The total number of annotations is O(mnptq) for verifying all abstract commonsense triples, which is also huge compared with the total number of original commonsense triples.
The errors are mainly due to the loss of contextualization within the original head events, as conceptualized head events with too high abstractness are likely to omit salient properties. For example, conceptualizing “watching a scary movie” as “watching movie” will lose the property “scary,” which further leads to a wrong abstract commonsense knowledge if the tail is “feel scared.” This also highlights the importance of verifying the plausibility of abstract commonsense knowledge that
heavily relies on both the contextualization brought by r, t and the conceptualization of the head event. Meanwhile, we observe that the models tend to make a neutral decision (plausibility score close to 0.5) when encountering the situation of conceptualizing an entire event as a concept with high-level abstractness. Indeed, they are more difficult abstract commonsense knowledge for machines to learn, as a higher level of abstractness leads to more possible instantiations and commonsense inferences.
E.2 Appliaction of CAT
Generative Event Conceptualization. The examples are shown in Table 16. Generated conceptualizations are generally plausible, given the head event as the context. Specifically, we observe that neural generators are more sensitive to the instance and its context, as heuristic matching may conceptualize “sleeping at night” and “having trouble sleeping at night” as “sleeping”. In contrast, neural generators can distinguish these two instances clearly by conceptualizing them as “sleep” and “sleep disorder”. One potential weakness of neural generators is that the generated conceptualizations lack diversity and novelty (Du et al., 2019; Wang et al., 2021), as they tend to be semantically close to the targeting conceptualizations in the training samples. Nevertheless, it still offers a reliable and simplified approach to performing contextualized conceptualization without tedious matching and human annotations. Such results also validate the reliability of our discriminative event conceptualization model, as the pseudo-labeled conceptualizations tend to be of high quality.
Commonsense Inference Modeling (COMET). Generations from COMET that are only trained on the ATOMIC subset, possibly augmented by abstract commonsense triples, are compared in Table 18. From these generations, we can observe that the abstract commonsense knowledge-aided COMET generator can generate tail events that are most plausible and generalizable compared with the one only trained on ATOMIC. It generally supports our hypothesis that abstract commonsense knowledge may implicitly help model situational commonsense inference, even without the instantiation step. In addition, this also validates that our automatically derived abstract knowledge is reliable and helpful, which also proves the reliability of our triple conceptualization discriminator.
E.3 Conceptualization by Large Language Models
With the recent advances in Large Language Models (LLMs), such as GPT3.5 (Brown et al., 2020; Ouyang et al., 2022) and ChatGPT (OpenAI, 2022), on various NLP tasks (Qin et al., 2023; Bian et al., 2023; Chan et al., 2023; Amin et al., 2023), we also aim to explore ChatGPT’s conceptualization ability through case studies. To do so, we investigate ChatGPT’s performance on three conceptualization tasks: discriminative event conceptualization, discriminative triple conceptualization, and generative event conceptualization, all of which are defined in Section 3. We randomly sample data entries from AbstractATOMIC and prompt ChatGPT with natural language commands to perform various tasks. The prompts used for performing these tasks are listed in Table 10. Specifically, we use OpenAI’s API6 to prompt ChatGPT and retrieve its generations.
The case studies for three tasks are presented in Table 12, Table 13, and Table 14, respectively. These demonstrate ChatGPT’s strong conceptualization abilities in both discriminative and generative manners. While ChatGPT can accurately determine most event conceptualizations and abstract commonsense knowledge, it still makes some mistakes. This highlights the value of training a performant discriminator through CAT, as it can effectively detect incorrect conceptualizations and implausible abstract commonsense knowledge. Additionally, ChatGPT tends to conceptualize instances using synonyms (Hagiwara et al., 2006) and hypernyms (Yu et al., 2020) and paraphrased or explained terms rather than higher-level concepts. This underscores the importance of our event conceptualization generator, which can generate precise, concise event conceptualizations. In conclusion, our work holds significant value in the realm of commonsense reasoning through conceptualization, particularly in light of the rise of large language models.
6The code for the model is gpt-3.5-turbo, and the date of access is May 2023.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Yes, in the limitation section on Page 10.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Yes, in the ethics statement section on Page 10.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Yes, in both sections on the first page.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Yes, as introduced in Section 3, Problem Definition, and Section 5, Experiments. There is also an additional explanation in Appendix A, Dataset Description.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Yes, all datasets are properly cited throughout the paper.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Yes, in the ethics statement section on Page 10, all datasets are shared via open-access licenses.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Yes, in the ethics statement section on Page 10, our use of existing artifacts is consistent with their intended use for research purposes.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Yes, in the ethics statement section on Page 10, the primary dataset is desensitized and anonymized.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Yes, in Appendix A, the dataset description.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, tables reporting statistics can be found in Section 3, problem definition, and appendix A, Dataset Description.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Yes, as shown in Section 5 and Appendix C.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Yes, parameters are reported in Table 11 in Appendices, and computational budgets are reported in Appendix C.1.2 and Appendix C.2.1.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Yes, in Appendix C.1.2 and Appendix C.2.1.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Yes, in Section 5 and Appendix C.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Yes, the usage of these packages is well-explained, and packages are cited throughout the Appendix.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Yes, expert annotations are conducted in Section 5.2, Generative Event Conceptualization. Details are introduced in Appendix C.2.2.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Yes, in Appendix C.2.2.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Yes, in Appendix C.2.2.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Yes, in Appendix C.2.2.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Not applicable.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Not applicable.

heading: Future Work
text: In addition, we only evaluated the power of the acquired abstract commonsense knowledge on the commonsense knowledge generation task (COMET), while other commonsense reasoning tasks remain future works (Wang et al. , 2023a), such as COLA (Wang et al. , 2023b), CommonsenseQA (Talmor et al. , 2019, 2021), SocialIQA (Sap et al. , 2019b), Winograd Schema Challenge (Levesque et al. , 2012), PIQA (Bisk et al. , 2020), Abductive Commonsense Reasoning (Bhagavatula et al. , 2020), and Winogrande (Sakaguchi et al. , 2020)