Complex Reasoning over Logical Queries on Commonsense Knowledge Graphs (2024)

Abstract: Event commonsense reasoning requires the ability to reason about the relationship between events, as well as infer implicit context underlying that relationship. However, data scarcity makes it challenging for language models to learn to generate commonsense inferences for contexts and questions involving interactions between complex events. To address this demand, we present COM (COMplex COMmonsense), a new dataset created by sampling multi-hop logical queries (e.g., the joint effect or cause of both event A and B, or the effect of the effect of event C) from an existing commonsense knowledge graph (CSKG), and verbalizing them using handcrafted rules and large language models into multiple-choice and text generation questions. Our experiments show that language models trained on COM exhibit significant improvements in complex reasoning ability, resulting in enhanced zero-shot performance in both indomain and out-of-domain tasks for question answering and generative commonsense reasoning, without expensive human annotations.1

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11365–11384 August 11-16, 2024 ©2024 Association for Computational Linguistics
Our experiments show that language models trained on COM2 exhibit significant improvements in complex reasoning ability, resulting in enhanced zero-shot performance in both indomain and out-of-domain tasks for question answering and generative commonsense reasoning, without expensive human annotations.1

heading: 1 Introduction
text: Large language models struggle to effectively perform reasoning when presented with complex tasks, such as reasoning about multiple events and their relationships. This shortcoming is due to both the inherent difficulty of reasoning over multiple pieces of information, as well as a lack of adequate-scale, supervised training datasets for learning (Zhao et al., 2023). Unfortunately, complex and multihop commonsense reasoning benchmarks (Gabriel et al., 2021) are both technically challenging and financially expensive to curate. Consequently, previous efforts either constructed datasets (a) with simpler reasoning structures, such as single-hop
∗ Work done during internship at EPFL. 1Code and data are available at https://github.com/
tqfang/complex-commonsense-reasoning
chains (Mostafazadeh et al., 2020), (b) using distant supervision based on one-hop inference (Gabriel et al., 2021), or (c) with human-annotations, but at a relatively small scale (Ravi et al., 2023).
To alleviate this training data bottleneck, recent works have explored extracting and formulating questions from existing CommonSense Knowledge Graphs (CSKGs; Hwang et al., 2021), which store commonsense triples. However, using CSKGs to produce high-quality reasoning datasets poses several challenges. First, while the shared entities in commonsense triples encode a complex, interconnected graph structure, the sparsity of this structure limits the number of potential questions that encode more than one reasoning hop (Sap et al., 2019b; Kim et al., 2023). Second, triples in CSKGs are represented in a context-free manner, such as the event “PersonX gets tired of it” in Fig. 1, yielding ambiguous (and sometimes incorrect) human annotations in the CSKG, e.g., ATOMIC (Sap et al., 2019a) has an error rate of over 10%. These errors propagate when triples are naively combined to construct reasoning questions. Finally, also because triples in CSKGs are represented in a context-free manner, additional context must be added to make questions
11365
fluent, a problem exacerbated in multi-hop settings where the entities of multiple reasoning hops must be coherently verbalized together.
In this paper, we construct COM2 (COMplex COMmonsense), a novel commonsense reasoning dataset using multi-hop queries in commonsense knowledge graphs to construct question answer pairs requiring complex narrative reasoning. To build this dataset, we use conjunctive logical queries (Hamilton et al., 2018), a subset of FirstOrder Logical queries that use existential quantifiers and conjunction. The multi-hop projection operation involves inferring hidden contexts, while the intersection operation enables reasoning among multiple events, encompassing common cause or effect, and abduction. For example, in Fig. 1, an intersection of two triples can be verbalized to a short narrative, and the process of inferring the common tail can be seen as an abduction of the hidden cause between the two heads.
To address the challenges above, we propose to first densify the CSKG to merge nodes with high semantic similarity, increasing the connectivity of the graph. Then, we use an off-the-shelf plausibility scorer to filter out low quality triples, avoiding error propagation as we construct more complicated queries. Finally, we verbalize the queries to a natural language context with handcrafted rules and Large Language Models to derive coherent and informative narrative contexts for our questions. Our final COM2 dataset comprises 790K questionanswer pairs (both with multiple-choice and generative answer settings), including 1.3K examples that we manually verify for evaluation.
Our results demonstrate the challenges faced by even powerful LLMs and supervised question answering models on the COM2 dataset, underscoring the difficulty of performing complex multihop reasoning. Moreover, fine-tuning question answering models and generative commonsense inference models on COM2 leads to substantial improvements across eight commonsense reasoning datasets, showing the efficacy of our framework for boosting commonsense reasoning ability.
To conclude, our contributions are three-fold. First, we present a pipeline for sampling and verbalizing complex logical queries from CSKGs, to form a complex commonsense reasoning benchmark, COM2, with minimal human effort. Second, we benchmark the complex reasoning ability of various state-of-the-art language models and question answering models on COM2. Finally, we validate
the benefit of fine-tuning on COM2 on eight zeroshot commonsense reasoning datasets.

heading: 2 Background and Related Work
text: Complex Logical Queries Recent years have witnessed significant progress in reasoning on onehop relational data (Bordes et al., 2013; Sun et al., 2019; Lin et al., 2023). In addition to one-hop reasoning, further works have explored handling complex logical structures, involving reasoning on unobserved edges and multiple entities and variables (Ren et al., 2020; Wang et al., 2021, 2023b; Bai et al., 2023a). In this paper, we focus on conjunctive logical queries (Hamilton et al., 2018), a subset of first-order logic that is defined with logical operators such as existential quantifiers ∃ and conjunctions ∧. Conjunctive logical queries require a set of anchor entities, V , a unique target entity V? representing the answer to the query, and a set of existential quantified variables V1, · · · , Vm, and are defined as the conjunction of literals e1, · · · , en:
q = V?, ∃V1, · · · , Vm : e1 ∧ e2 ∧ · · · ∧ en (1)
where ei is an edge involving variable nodes and anchor nodes, satisfying ei = r(vj , Vk), Vk ∈ {V?, V1, · · · , Vm}, vj ∈ V, r ∈ R, or ei = r(Vj , Vk), Vj , Vk ∈ {V?, V1, · · · , Vm}, j ̸= k, r ∈ R. R is the set of relations defined in the KB.
Previous efforts on answering logical queries on knowledge graphs focus on constructing box embeddings (Ren et al., 2020), embeddings based on beta distributions (Ren and Leskovec, 2020), particle simulations (Bai et al., 2022), and computation tree optimization (Bai et al., 2023b). Other related works focus on leveraging two-hop projection and intersection queries in ConceptNet to improve commonsense question answering (Guan et al., 2023), inferring missing entities in verbalized complex queries on factual knowledge graphs (Ding et al., 2023), and developing an LLM agent for complex operators within the KG (Jiang et al., 2024). Instead of relying on embeddings or limited query types for matching synthetic logical queries, we leverage the concept of logical queries to effectively acquire complex reasoning data from CSKGs with minimum human efforts.
Complex Commonsense Reasoning Recent advances in commonsense reasoning have been driven by the construction of human-annotated (Speer et al., 2017; Sap et al., 2019a; Hwang et al.,
2021; Jiang et al., 2021; Mostafazadeh et al., 2020; Krishna et al., 2017; Shen et al., 2024) and humanvalidated (West et al., 2022; Gao et al., 2023) CommonSense Knowledge Graphs (CSKG). A common approach to create challenges for commonsense reasoning involves constructing tasks in the form of question-answering (Talmor et al., 2019; Sap et al., 2019b), knowledge base completion (Malaviya et al., 2020; Yang et al., 2023) and population (Fang et al., 2021b,a), grounding (Gao et al., 2022), and daily dialogue (Kim et al., 2023), based on CSKGs. However, most of those previous benchmarks are based on one-hop triples.
In contrast, real-world situations in narratives usually involve more complicated reasoning across multiple events, sentences, and paragraphs (Schank and Abelson, 1975). Previous works learn representations of narrative chains (Chambers and Jurafsky, 2008; Pichotta and Mooney, 2014) and draw inferences (Fang et al., 2022; Yuan et al., 2023). To address more complex paragraph-level or multievent reasoning, ParaCOMET (Gabriel et al., 2021) proposed to pre-train on distantly supervised onehop paragraph-level commonsense inferences, and COMET-M (Ravi et al., 2023) was fine-tuned on a crowdsourced corpus focusing on reasoning on multiple events. Instead of crowdsourcing or using language models to distill complex inferences, we provide narrative-level inference by verbalizing complex logical queries over CSKGs, to effectively acquire grounded inferences at scale.

heading: 3 Methodology
text: In this section, we introduce the construction details of COM2, including the pre-processing, sam-
pling, and verbalization of complex queries, as well as the details of human annotations. An ovewview of the pipeline is presented in Fig. 2.

heading: 3.1 Pre-processing
text: We use ATOMIC2020 (Hwang et al., 2021), a comprehensive Commonsense Knowledge Graph covering everyday social, physical, and event-level knowledge, as the base CSKG. Before sampling queries, we address the sparsity and quality issues first.
Sparsity CSKGs are usually highly sparse compared to factual KGs due to the diversity and scale of commonsense (Malaviya et al., 2020), resulting in many isolated nodes that can hardly be sampled as part of a complex query. To alleviate this issue, we develop a set of rules and use sentence embedding similarity to merge nodes in the CSKG, leading to 22.4% of nodes being merged and an average degree increase of 25.3%. In the final query sampling process, the number of 2p paths increased from 7,382 to 405,492, and the number of 2i queries rose from 1.43M to 2.06M.
Quality The error rate of CSKGs (e.g., ATOMIC has an error rate of ∼10%) can be problematic when we consider the intersection and projection of more than two triples (errors in a single triple could propagate to many multi-hop queries). We use an off-the-shelf plausibility scorer Vera (Liu et al., 2023), a T5-based scorer fine-tuned on 2 CSKGs and 19 QA datasets, to score every triple in terms of commonsense plausibility (between 0 to 1). We filter out triples (∼10%) with a plausibility score lower than 0.5, the threshold provided in Liu et al. (2023) for plausible statements.

heading: 3.2 Query Sampling
text: The query structures that we study are visualized in Fig. 3. Following Ren et al. (2020), we use projections (1p, 2p) and intersections (2i, 3i) as training queries, and leave complex queries ip and pi as zero-shot evaluation queries. To examine scenarios involving negation and differentiate them from regular 2i queries, we use the term “2i-neg” to represent 2i queries where one of the relations is “HinderedBy”. In this formulation, multi-hop projection involves inferring hidden reasoning contexts, while intersection operations require reasoning about complex interactions between events.
Given a query structure, we use pre-order traversal to sample free variables and anchor entities starting from an answer entity. We sample predecessors uniformly based on (relation, entity) pairs. During sampling, to avoid over-sampling on nodes with extremely high degree, we empirically set a cut-off degree T = 10 to only sample from top T neighbors of a node scored by Vera. In the end, we conduct a post-order traversal starting from the anchor entities to find all the answers of the query, in addition to the starting answer entity.
Distractor Sampling We sample 4 additional candidate distractors for each query, where 2 of them are randomly sampled across the whole CSKG, and 2 of them are sampled from the neighbors of the anchor entities that are not the answers to the whole query, represented as adversarial negative examples. When fine-tuning a question answering model, the negative examples are used as syn-
thetic question answering pairs for training. In the evaluation set, these candidate negative examples, together with the sampled answer, are manually annotated to form a gold evaluation set.

heading: 3.3 Verbalization
text: CSKGs are constructed in a context-free manner. To make the logical queries on such context-free triples more human-interpretable, we introduce an additional step of verbalizing the anchor entities to a narrative, to effectively acquire fluent and plausible narrative-inference pairs.
Anchor Entity Verbalization We consider a rule-based verbalizer and a ChatGPT-driven verbalizer. In the rule-based verbalizer, we add a discourse marker between the two or three anchor entities depending on the semantics of the query relations. For example, a simple situation would be adding an “and” or “then” between two anchor entities in a 2i query. To make the query more human-understandable, we consider using ChatGPT to synthesize necessary contexts to make the query an actual narrative. We include the detailed rules for adding discourse connectives, and prompts for using ChatGPT to verbalize complex queries in Appx. §A.3.
Relation Verbalization The multiple relations in complex queries can be deterministically converted to a question using the natural language descriptions of the relations, presented in Appx. §A.3.

heading: 3.4 Human Annotation
text: To support reliable automatic evaluation, we formalize the problem of complex commonsense reasoning as a multi-choice question answering task, with one true answer, three distractors, and a fifth option indicating “None of the answers are correct”. We crowdsourced the answers using Amazon Mechanical Turk (AMT). The workers are given the verbalized query as the context, the verbalized relations as the question, and the sampled (negative) answers. If no sampled answers are correct, then the worker is asked to select an additional “None of the answers are correct” option. If the verbalization itself does not make sense, the worker can also select another option “The context doesn’t make sense or is meaningless” and we discard the example. Each question is annotated by three workers. The workers are paid on average 16 USD per hour. Our final dataset consists of ∼782k training examples and 1317 manually-validated evaluation examples.
Quality The overall per-option inter-annotator agreement is 78%, and the Fleiss kappa is 0.445, indicating moderate agreement. Among 1.3K verified examples, 4.7% were labeled as incorrect contextualization. The likelihood that a sampled answer is the correct response to the contextualized question is 52.1%. For randomly sampled negative examples and one-hop neighbors, the plausibility rate is 23.5%, notably lower than the sampled an-
swers. The authors of this paper manually checked the examples where the IAA between three annotators is lower than 0.6 and fixed the answers to ensure quality. A similar distribution is expected for the training set. Another thing to note that even though the training set is silver-standard, language models fine-tuned on it can autonomously identify patterns and acquire valuable insights from a large number of complex queries, resulting in improved reasoning performance, which will be shown in the next section.
More details can be found in Appx. §A.

heading: 4 Experiments
text: We conduct experiments on the evaluation set of COM2, formulated as a Multi-Choice Question Answering (MCQA) task. Specifically, we examine the performance of state-of-the-art off-the-shelf language models on COM2, and also study the effect of training a question answering model on the distantly supervised training set of COM2.

heading: 4.1 Setup
text: We use popular API-based and open-source LLMs as baselines. Following the standard practice of prompting LLMs for QA (Robinson et al., 2022), we initialize a prompt that takes “[Context] [Question] [Options]” as the input and ask the model to only output the associated symbol (e.g., ‘A’) in the QA pair as the prediction. For open-source lan-
guage models like Flan-T5 and Llama2, we use the same prompt, and compute the logits received by each of the options in the first prediction token.
We also study the effect of fine-tuning a questionanswering model on the synthetic training queries discussed in §3.2. We follow the pipeline by HyKAS (Ma et al., 2021), which fine-tunes language models on QA pairs synthesized from onehop knowledge in CSKGs, and extend it to complex queries. For one-hop (1p) triples, the head and relation are transformed into a question with pre-defined prompts. For complex queries, the verbalized queries (as illustrated in §3.3) are regarded as the context, and questions are also transformed with a different prompt template depending on the relations. The tails to the one-hop triple or the sampled answer to the query are regarded as the correct answer, and the negative examples are randomly sampled across the whole CSKG following a keyword overlapping filtering (Ma et al., 2021; Wang et al., 2023a). We use DeBERTa-v3-large as the backbone encoder.2

heading: 4.2 Results and Analysis
text: Our results are presented in Tab. 1. We observe that Chain-of-Thought (CoT) improves reasoning performance, as it allows the model to first in-
2We refer readers to Appx. §B for detailed implementations and prompt templates.
duce the causes or effects of individual events in intersection-based queries (2i and 3i), or induce hidden variables in projection-based queries (2p as in Fig. 3). Adding eight-shot exemplars (consisting of 2i, 2i-neg, and 2p queries) further improves performance among prompting baselines.
For models fine-tuned on complex queries using HyKAS and CAR, we observe that the synthetic training pairs, despite lacking manual annotation, serve as valuable distant supervision signals. They enhance the complex reasoning capability of HyKAS and CAR, surpassing the performance of the 8-shot GPT-4 model with CoT by 6%. CAR + COM2 also outperforms the 11B version of UnifiedQA-v2 and Flan-T5, which are both fine-tuned on numerous (commonsense) question answering datasets, by 9% and 3%, respectively.

heading: 5 Downstream Evaluation
text: In addition to benchmarking Complex Commonsense Reasoning, we also study the effect of leveraging COM2 as training data to generalize to other downstream commonsense reasoning tasks. As tasks, we use zero-shot CommonSense Question Answering (CSQA), and Generative Commonsense Inference, including one-hop, multi-event, and paragraph-level settings.

heading: 5.1 Commonsense Question Answering
text: Setup The task of zero-shot commonsense QA involves selecting the most plausible option for commonsense questions without training on examples from the benchmark dataset. We directly leverage the model we trained in §4, the DeBERTa-v3-largebased model fine-tuned on synthetic question pairs from both ATOMIC and COM2, and check the performance on five popular commonsense question answering datasets: Abductive NLI (aNLI; Bhagavatula et al., 2020), CommonsenseQA (CSQA; Talmor et al., 2019), PhysicalIQA (PIQA; Bisk et al., 2020), SocialIQA (SIQA; Sap et al., 2019b), and WinoGrande (WG; Sakaguchi et al., 2021). As baselines, we consider the same methods, HyKAS (Ma et al., 2021) and CAR (Wang et al., 2023a), but use other CSKGs as training sets. In Tab. 2, ATM-10X refers to ATOMIC-10x from West et al. (2022), and ATMC refers to the training data from CAR (Wang et al., 2023a) which is augmented from ATOMIC with conceptualization.
Results and Analysis We report model performance in Tab. 2. We observe the inclusion of COM2 and one-hop triples from ATOMIC as training data for CAR and HyKAS yields significant improvements in question answering ability. Notably, the combination of CAR and COM2 achieves the highest performance among all models, surpassing even ChatGPT and GPT-4, despite having a parameter size at least two orders of magnitude smaller.
Notably, when using CAR as the base model,
training on COM2 leads to the highest performance gain of around 1.8% for a-NLI. When evaluating on a-NLI, which includes instances of abductive reasoning, the model may be helped by learning from 2i queries where one relation represents cause and the other represents effect (abduction examples in Fig. 1 and Fig. 4). Meanwhile, the performance on WinoGrande was adversely affected, likely because Winogrande primarily focuses on identifying distinguishing features of entity pairs. The benefits from learning event-event interactions from COM2 may not transfer well to this setting.

heading: 5.2 Generative Commonsense Inference
text: Setup We study generative commonsense inference as an additional evaluation task. We include multi-event commonsense generation (COMETM; Ravi et al., 2023) and paragraph-level commonsense generation (ParaCOMET; Gabriel et al., 2021) as two out-of-domain evaluation tasks. We also include the vanilla COMET (Bosselut et al., 2019) as an additional in-domain evaluation, which focuses on 1p queries that require generating the tail given head and relation as the input. We also conduct experiments on the generative sub-task of COM2, where verbalized context and question inputs are used to inferences. The annotated ground answer options are used as references.
For the (distantly) supervised learning baselines, we fine-tune GPT-2-large on the annotated multievent inference dataset (MEI) from Ravi et al.
(2023) and distantly labeled PCD dataset from Gabriel et al. (2021) as a reference. In our zeroshot learning setting, we study the effect of finetuning COMET (GPT-2-large) on ATOMIC and different query types of COM2. We also study finetuning an LLM, Llama2-7b, by converting triples and queries to an instruction-tuning format, following the prompt template in §3.3 and Appx. §B.2. We leverage the framework of Chen et al. (2023)3 to fine-tune Llama2-7b. We fine-tune on a mixture of different query types as detailed in the Training Data column. We present the performance results of models fine-tuned on either the annotated or distantly supervised training set for both tasks as reference benchmarks. Specifically, we use MEI for COMET-M and PCD for ParaCOMET. To ensure diversity and prevent overfitting to common tails, complex queries are selected using an n-gram based diversity filter (Yang et al., 2020).
Results and Analysis We present the results in Tab. 3. Compared to models fine-tuned solely on one-hop triples, COMET models fine-tuned on additional complex queries demonstrate enhanced generative commonsense inference capabilities for multi-event and paragraph-level scenarios. When comparing different query types, fine-tuning solely on 2i queries yields the most significant improvement in reasoning capability, likely because 2i queries provide more explicit reasoning signals compared to 2p queries, which can be ambiguous due to the large candidate space of the hidden event. For example, the average number of answers for 2p queries is 7.93, compared with 1.09 for 2i queries. In addition, the answers to 2i queries exhibit greater diversity than 3i queries, as the CSKG is sparse and provides a limited number of distinct tails for sampling 3i queries compared to 2i queries.

heading: 6 Analysis & Discussion

heading: 6.1 Ablation Study
text: We analyze the impact of various data filters, query types, and verbalization methods on generative inference within COM2. Detailed results can be found in Tab. 4.
Filtering We include two types of filters, a Verabased plausibility filter and a diversity filter. Evaluating the performance of generative commonsense inferences on COM2, we examine the impact of removing both filters while employing GPT2-Large
3https://github.com/epfLLM
as the backbone model. Removing the plausibility filter results in a significant performance decline, highlighting its critical role. On the other hand, the diversity filter exhibits a minor positive influence on enhancing performance.
Type of Queries We investigate the impact of training our models on different types of logical queries. The model trained only on 1p and 2p queries does not generalize well to other query types such as pi and ip, leading to a worse performance than the model trained on all query types. However, according to Tab. 1 and Tab. 3, models trained on only 2i queries generalize better to downstream commonsense reasoning tasks, potentially indicating that multi-event reasoning in most existing commonsense benchmarks focus on intersection more than projection.
Verbalization We investigate the effect of using a rule-based verbalizer or ChatGPT-enabled verbalizer to generate COM2 contexts. Using ChatGPTverbalized queries leads to better downstream performance on both PCD and COM2. In COM2, the presence of ChatGPT-verbalization intuitively improves performance since the training context aligns with the evaluation set’s format. On the other hand, the context in the PCD dataset is long and
comprised of five sentences. Verbalization not only adds more contexts to the training but also aligns better with the PCD format.

heading: 6.2 Error Analysis
text: We present a human-annotated quality evaluation of the Llama-7b-based model on the generation sub-task of COM2. To ensure diverse coverage of query types, we randomly sampled 60 queries, with 10 from each of the 6 types. Manual inspection revealed a common error where the generated output was partially correct, either providing the answer to one of the triples in an intersection query or only the one-hop answer instead of the two-hop answer in 2-projection (2p) queries. Tab. 5 includes the number of such ‘1-hop’ partially correct answers. Our results demonstrate that the zero-shot Llama model already produces 26 out of 60 plausible inferences. Fine-tuning the model on one-hop ATOMIC further increases the number of plausible generations while more frequently generating inferences that are one-hop correct. Moreover, fine-tuning on the synthetic training set of COM2 significantly improves the model’s ability to generate complex commonsense inferences and reduces the occurrence of partially correct answers. We provide case studies in Appx. §D.

heading: 7 Conclusion
text: In this paper, we leverage the concept of conjunctive logical queries to create a complex commonsense reasoning dataset derived from CSKGs. The dataset, COM2, comprises a human-annotated evaluation set and a distantly supervised training set without further annotations. Our experiments highlight the challenging nature of complex commonsense reasoning that involves multiple events or multi-hop scenarios, even for advanced language models such as GPT-4. Additionally, we train question answering models and generative commonsense reasoning models using COM2. The re-
sults show significant improvements across eight diverse downstream commonsense reasoning tasks, highlighting the potential of leveraging CSKGs to acquire complex reasoning signals inexpensively, without relying on extra human effort.

heading: Acknowledgement
text: Yangqiu Song was supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20), and the GRF (16211520 and 16205322) from RGC of Hong Kong. Yangqiu Song thank the support from the UGC Research Matching Grants (RMGS20EG01-D, RMGS20CR11, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08). We thank the support from the Tencent AI Lab Rhino-Bird Focused Research Program. We also gratefully acknowledge the support of the Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29), the EPFL Science Seed Fund, the EPFL Center for Imaging, Sony Group Corporation, and the Allen Institute for AI.
Limitations
Data Construction The construction of COM2 relies on sampling complex logical queries from existing CSKGs, which requires addressing sparsity, quality, contextualization issues. Despite conducting normalization and filtering, there may still be missing links within ATOMIC and mislabeled or ambiguous triples, which limits the quality of our sampled queries. Future works can focus on deriving complex queries from CSKGs with better quality and more diverse semantics, which should also have higher density, such as on ATOMIC-10x, NovATOMIC (West et al., 2023).
Evaluation In the context of generative commonsense reasoning, we employ lexical-overlap based automatic evaluation metrics to assess the performance of the model in a scalable manner. However, since each query typically has 1 to 3 gold references on average, this type of evaluation may not accurately capture the true plausibility of commonsense inferences, which is inherently open-ended. To address this limitation, we have supplemented the automatic evaluation with human annotation on a subset of sampled queries, but this approach is not scalable.
Ethical Considerations
We sample the data from ATOMIC2020, which is an open-source commonsense knowledge graph that may contain biases around gender, occupation, and nationality (Mehrabi et al., 2021). When constructing COM2, these biases may propagate if biased triples are sampled in a complex query that becomes of the training set. We collected 1.3k inferences through crowdsourcing. The participants were compensated with an hourly wage of 16 USD, which is comparable to the minimum wages in the US. The qualification was purely based on the workers’ performance on the evaluation set, and we did not collect any personal information about the participants from MTurk.

heading: A Additional Details on Data Construction
text: In this section, we provide additional details to node normalization, plausibility filter, verbalization, and human annotations. The overview of our construction framework is presented in Fig. 2.
A.1 Nodes Normalization (Dealing with Sparsity)
To alleviate the sparsity issue, we first normalize the tail entities with simple rules similar with that in Dense-ATOMIC (Shen et al., 2023) and CKBP (Fang et al., 2021a). In ATOMIC, heads are pre-defined complete sentences (for example, “PersonX says sorry”) while tails are usually short phrases without a subject (for example, “to say sorry”). This discrepancy produces many duplicated nodes and make the graph sparser. We develop simple rules to add “PersonX” or “PersonY” in front of the tails to make them a complete sentence, if the tail does not have a subject. This process merged 3.7% nodes together.
Second, as the nodes in ATOMIC are free-text, some nodes with the same semantic meaning are represented as separated nodes due to some minor annotation distinctions and errors, e.g., “PersonX buys a ticket” versus “PersonX buys a ticket .”. These discrepencies can be addressed using embedding similarities (Wu et al., 2023). We use a state-of-the-art sentence embedding model4, to merge nodes with cosine similarity score over 0.95. In this process, 20.0% nodes are merged together and the average degree increases by 25.3%.
A.2 Data Filtering
Plausibility Filter We verbalize a (h, r, t) triple from ATOMIC using the default template as provided in Hwang et al. (2021). For example, (PersonX repels PersonY’s attack, xAttr, brave) would
4https://huggingface.co/sentence-transformers/all-mpnetbase-v2
be transformed to a declarative statement “If PersonX repels PersonY’s attack, then PersonX is seen as brave”. To obtain a plausibility score, we input the statement into the Vera-5B model. 0.5 is used as the threshold to draw a boundary between plausible and implausible statements. We perform a manual inspection on the triples scored by Vera and randomly select 40 samples for three plausibility score intervals. Among these, we find that 4/40 triples are plausible when the Vera scores range from 0 to 0.1. 13/40 triples are considered plausible within the score range of 0.2 to 0.25. Furthermore, we identify 20/40 triples as plausible when their plausibility scores hover around 0.5, when most of the triples are quite ambiguous. By setting the filter threshold as 0.5, we filter out around 14% triples that are of a relatively lower quality.
Diversity Filter To prevent overfitting to common tails, we conduct a diversity-based filter to acquire diverse queries for training. We take inspirations from G-DAUG (Yang et al., 2020), to use a simple greedy algorithm to iteratively select training data, which has been proven useful for selecting augmented data. To be more specific, for each unique answer, we adopt an iterative approach to select the verbalized query that contributes the highest number of unique 1-gram terms to an ongoing vocabulary constructed for each answer. We select top-20 queries for each unique answer entity.
A.3 Verbalization
Query Verbalization We employ two methods to verbalize complex queries: a rule-based method and a ChatGPT-based method.
In the case of 2i and 3i queries, the rule-based method typically involves inserting an “and” between the anchor entities. However, if the query suggests a specific chronological order between the two events, we use “then” to connect the events. For instance, in 2i queries where one triple is (V1, xEffect, V?) and the other is (V2, xIntent, V?), it implies that V? serves as the effect of V1 and the intermediate hidden cause of V2. In this scenario, V1 should occur before V2. Therefore, the verbalization would be “V1 then V2”.
For ChatGPT verbalization, we present the system instructions for verbalizing different kinds of queries in Tab. 7. Then, we generate the verbalized contexts with six exemplars that are manually annotated. In the system instruction, we also ask ChatGPT to output “NA” if the given anchor en-
tities are totally irrelevant or too ambiguous. We filter out those queries where the output is “NA”.
For example, to better interpret the query in Fig. 1, we need to take into consideration both the relations of interest and the anchor entities. The query asks about the effect of the first event and what causes (intention) of the second event, which is inherently represents abductive reasoning. This requires the second event to happen before the first event, to derive reasonable abduction. In this sense, a natural rule of verbalizing the query would be adding a discourse connective “after” to convert the query to “After PersonX gets tired of it, PersonX goes skydiving”. However, the verbalized query may still be ambiguous without additional context. To make the verbalized context more informative and human-understandable, we take advantage of Large Language Models (i.e., ChatGPT) to add additional context to compose the query to a narrative.
Relation Verbalization We use conversion rules and pre-defined templates to compose questions based on the relations in the queries. Based on the definition of each commonsense relation (Hwang et al., 2021), we use the templates in Tab. 8 to verbalize each relation. In terms of complex queries, we use the conversion rules in Tab. 9 to convert the query to a question.
Person Names To make the context more natural, we replace PersonX, PersonY, PersonZ in the context to names randomly sampled from the 2021 public US social security application name
registry5.
A.4 Human Annotation
We introduce the details of the annotation process in this subsection.
Worker Selection We have a qualification test to select eligible workers for the main task. We prepare six pre-selected 2i queries of different types, including (negated) common effect, (negated) common cause, common attribute, and abduction. Only Master annotators are eligible for participating the qualification. We compare the pair-wise annotation accuracy between each annotator and the gold answer annotated by the authors of the paper, and select those who have at least 85% agreement as qualified workers. After selection, we pick 53 worker out of 120 participants in the qualification round.
5https://catalog.data.gov/dataset/baby-names-fromsocial-security-card-applications-national-data
Annotation Interface A snapshot of the annotation interface is presented at Fig. 5. In addition, we have provided comprehensive instructions along with detailed examples to guide the annotators throughout the annotation process. To ensure their understanding, we require annotators to confirm that they have thoroughly read the instructions by checking a checkbox before the annotation task. We also manually checked the performance of the annotators along with the annotation process and gave feedbacks based on common errors. For example, typical errors include mistakenly regard the one-hop answer as correct instead of fully considering the multi-hop context.
Post-processing To aggregate the annotation result, we randomly sample one option that is labeled
as plausible by majority voting as the final positive answer, and sample three negative options and distractors. If there are no options labeled as plausible, then the correct answer is “None of the answers are correct”. If there are less than three options labeled as negative, we manually add one or two negative examples to match the number. To improve the quality, after crowdsourcing, the authors of this paper manually checked the QA pairs with an IAA lower than 0.6, and resolve the disagreements manually.

heading: B Additional Details of Experiments
text: B.1 Implementation Details of the Question Answering Models
We follow the pipeline in HyKAS (Ma et al., 2021) and CAR (Wang et al., 2023a) Let C represent the original context, which is the head entity for 1p triple and the verbalized context for complex queries, Q represent the question verbalized from the anchor relations, and (A1, A2, ...) be the list of options. We first concatenate C, Q, and an answer option Ai together via natural language prompts following the order of “C Q Ai” to generate input sequences (T1, T2, ...). We then repeatedly mask out one token at a time to calculate the masked language modeling loss.
S(T ) = − 1 n
n∑
i=1
logP (ti|..., ti−1, ti+1, ...) (2)
We then compute the marginal ranking loss based on Equation 3, where η represents the margin and y is the index of the correct answer.
L = 1 m
m∑
i=1,i ̸=y max(0, η − Sy + Si) (3)
We train the DeBERTa QA model for 1 epoch with a learning rate of 5e-6 and a linear learning rate decay. The checkpoint that yields the best performance on the synthetic validation set in CAR (Wang et al., 2023a) or HyKAS (Ma et al., 2021) is selected as the final model. During evaluating, we select the option that yields the lowest score as the final prediction.
We provide the prompt templates for each model in Tab. 10.
B.2 Implementation Details of Generative Commonsense Inference Models
The training and evaluation of GPT2-based model is based on the paradigm defined in COMET (Bosselut et al., 2019). The input of onehop ATOMIC triples is serialized to “h r” and the expected output is t, where (h, r, t) forms a triple
in the CSKG. The input of 2p queries, (h, r1, V ) and (V , r2, V?), are serialized as “h r1 r2” and the expected output is V?. The input of 2i queries, which includes (h1, r1, V?) and (h2, r2, V?), is serialized as “h1 h2 r1 r2” with the expected output as V?. All models are fine-tuned for 3 epochs with a batch size of 32, a learning rate of 1e-5, a linear learning rate decay. The last checkpoint is taken as the final model.
For Llama2, we follow the standard instruction tuning procedure and use the pipeline provided by Chen et al. (2023). We train the model with a batch size of 32, learning rate of 1e-5, and linear learning rate decay. We take the final checkpoint as our model to make prediction.
The whole list of prompt templates that we use is presented in Tab. 11.

heading: C Additional Analysis
text: Differences from ParaCOMET and COMET-M In ParaCOMET, the task involves providing a narrative as input, requiring the model to determine the commonsense causes or effects of a specific sentence within the context. To generate training
data, a single-hop COMET model fine-tuned on ATOMIC is employed to create synthetic inferences. These inferences are generated solely based on the target sentence and the desired relation, without accessing the whole context. The resulting onehop synthetic inferences are then utilized as distant supervision signals during the fine-tuning process for ParaCOMET.
COMET-M utilizes a context consisting of a sentence containing multiple events. Unlike from a sentence level, COMET-M focuses on generating commonsense inferences based on a specific event within the sentence. T his fine-grained approach enables more precise and detailed commonsense reasoning.
In contrast, our complex commonsense reasoning benchmark introduces additional complexities compared to ParaCOMET and COMET-M. Besides the complex structures in the context that involves multiple events, the desired relation or question involves multi-hop reasoning as well. For instance, rather than focusing on the cause of a single sentence or event, COM2 explores questions related to common causes, effects, attributions of multiple events, and two-hop inferences. This distinctive formulation sets our work apart and poses a greater challenge for LLMs to effectively reason and provide accurate responses.
Results of the Ablations We present the results of the ablation study in Tab. 4.
C.1 Difficulty of Different Query Types
The results in Tab. 1 showed that performance varied depending on the evaluation query types. Interestingly, pi queries exhibited a significantly higher success rate compared to other query types, particularly ip queries, considering both pi and ip involve a single free variable and both intersection and projection operations. We present two perspectives to explain this phenomenon. First, the limited availability of sampled pi queries restricts the diversity of the data. Out of all the queries sampled from the development set of ATOMIC2020, only 4k are pi queries, while there are 12k ip queries and 598k 2i queries. This paucity of pi queries contributes to a lack of variety. Moreover, within these 4k pi queries, the number of unique answers is limited to 459, indicating a limited range of possible responses. As a result, models fine-tuned on ATOMIC can generate answers to pi queries more easily, given that most of them consist of nodes
with high degrees. Second, the chances of the sampled answer is actually the correct answer to pi queries (67.8%) is significantly higher than other query types (e.g., 47.2% for ip). This is also a result of the first reason, as the answers to the sampled queries are limited to nodes with high degrees, which are usually events with a broad meaning such as “PersonX gets better”.
Discussions on Further Applications of Complex Queries Intuitively, 2i queries can represent various scenarios such as common attribution, common effect, common cause, and abduction (when one relation pertains to effects and the other relates to cause), depending on the types of relations involved in the query. Besides, complex logical queries, particularly those involving intersection operations, are relevant to defeasible reasoning (Rudinger et al., 2020), where inferences can be weakened given new evidence. In the one-hop setting, tails are annotated in a context-free manner, considering only the most general cases. However, in intersection-based queries like 2i and 3i, additional anchor entities and relations act as specific constraints, narrowing down the inferences to a particular scope while disregarding other commonsense inferences in the context-free scenario. For instance, in the example from Fig. 1, other potential tails for (PersonX goes skydiving, xIntent) could include overcoming fear, seeking enjoyment, or achieving a personal milestone. Nevertheless, when constrained by another query (PersonX gets tired of it, xWant), the intentions related to fear, enjoyment, and fulfillment are weakened, and only the correct inference of “finding new things to do” remains.

heading: D Error Analysis
text: We present some error cases in Tab. 12. In general, a common error in both projection and intersection queries is that the generated answer can be only the one-hop answer instead of the correct answer that is multi-hop. For example, in the 2p case, “get a new job” is a direct intention of someone who updates his or her resume. However, the 2p query asks about the intention of the intention, which requires inducing the intention behind “get a new job”. In this sense, “to be financially independent” is more plausible inference. In the case of 2i queries, the error lies in the absence of inferential gaps between the context, where the generated answers become paraphrases of the events rather than being the result by any anchor entity. In the case
of ip, a common error for one-hop COMET is the generation of “None” for complex cases, indicating a deficiency in multi-hop reasoning capabilities.

heading: Future Work
text: Future works can focus on deriving complex queries from CSKGs with better quality and more diverse semantics, which should also have higher density, such as on ATOMIC-10x, NovATOMIC (West et al. , 2023). Evaluation In the context of generative commonsense reasoning, we employ lexical-overlap based automatic evaluation metrics to assess the performance of the model in a scalable manner. However, since each query typically has 1 to 3 gold references on average, this type of evaluation may not accurately capture the true plausibility of commonsense inferences, which is inherently open-ended. To address this limitation, we have supplemented the automatic evaluation with human annotation on a subset of sampled queries, but this approach is not scalable. Ethical Considerations
We sample the data from ATOMIC2020, which is an open-source commonsense knowledge graph that may contain biases around gender, occupation, and nationality (Mehrabi et al. , 2021). When constructing COM2, these biases may propagate if biased triples are sampled in a complex query that becomes of the training set. We collected 1. 3k inferences through crowdsourcing. The participants were compensated with an hourly wage of 16 USD, which is comparable to the minimum wages in the US. The qualification was purely based on the workers’ performance on the evaluation set, and we did not collect any personal information about the participants from MTurk