Commonsense Knowledge Graph Completion Via Contrastive Pretraining and Node Clustering
Year: 2023

Abstract:
The nodes in the commonsense knowledge graph (CSKG) are normally represented by free-form short text (e.g., word or phrase). Different nodes may represent the same concept. This leads to the problems of edge sparsity and node redundancy, which challenges CSKG representation and completion. On the one hand, edge sparsity limits the performance of graph representation learning; On the other hand, node redundancy makes different nodes corresponding to the same concept have inconsistent relations with other nodes. To address the two problems, we propose a new CSKG completion framework based on Contrastive Pretraining and Node Clustering (CPNC). Contrastive Pretraining constructs positive and negative head-tail node pairs on CSKG and utilizes contrastive learning to obtain better semantic node representation. Node Clustering aggregates nodes with the same concept into a latent concept, assisting the task of CSKG completion. We evaluate our CPNC approach on two CSKG completion benchmarks (CN-100K and ATOMIC), where CPNC outperforms the stateof-the-art methods. Extensive experiments demonstrate that both Contrastive Pretraining and Node Clustering can significantly improve the performance of CSKG completion. The source code of CPNC is publicly available on https://github.com/NUSTM/CPNC.

heading: 1 Introduction
text: Commonsense knowledge graphs (CSKG) have been widely used to build commonsense-grounded AI applications, such as question answering (Lv et al., 2020), visual question answering (Zhu et al., 2020), sentiment analysis (Li et al., 2022), dialogue system (Tu et al., 2022), etc. Commonsense knowledge graphs such as ConceptNet (Speer et al., 2017) and ATOMIC (Sap et al., 2019) provide a structured way of representing a commonsense concept, which consists of a head node, a tail node, and
∗*Corresponding author
the relation edge. Nodes in commonsense knowledge graphs are typically represented by free-form short text (word or phrase), resulting in many different nodes representing the same concept. Figure 1 shows a subgraph of ConceptNet, where nodes in the same color indicate the same concept. E.g., “have breakfast”, “take breakfast” and “eat breakfast” all express the concept of “eat breakfast”. This problem also results in a large number of missing edges between nodes, as illustrated by the dashed lines in Figure 1. E.g., “eat breakfast” and “prepare food” have a “HasPrerequisite” relation, but such relation is missing between “eat breakfast” and “cook food”.
On the one hand, node redundancy in CSKG makes different nodes of the same concept have inconsistent relations with other nodes. In CSKG representation and completion, it would be beneficial to make use of the latent concept information behind different nodes to help learn more semantic-general representations. However, this intuition was ignored by most of the existing work in
13977
CSKG completion. On the other hand, as analyzed by Malaviya et al. (2020), edge sparsity in CSKG limits information propagation in graph neural networks. Upon graph neural networks, researchers further incorporated pre-trained language model such as BERT to enhance the semantic representation of nodes (Malaviya et al., 2020; Ju et al., 2022; Wang et al., 2021). However, fine-tuning BERT is still imperfect in representing the commonsense knowledge graph, which is made up of linked nodes represented in a free-form short text.
To tackle the two issues, we propose a new CSKG completion framework based on the Encoder-Decoder architecture, which contains two core modules Contrastive Pretraining and Node Clustering (CPNC). Contrastive Pretraining is to alleviate the difficulty of node representation learning induced by edge sparsity. Through contrastive learning on positive and negative head-tail node pairs, the embedding distance between related nodes becomes closer, and that between unrelated nodes becomes farther so as to learn better node representations. Node Clustering aims to address the edge inconsistency issue caused by node redundancy. We cluster nodes with close semantic representations and take the mean vector as the latent concept representation for nodes in this cluster. To assist CSKG completion, the latent concept representation is fused with the node representation.
We evaluate our CPNC framework on two CSKG completion benchmarks, i.e., CN-100K and ATOMIC. The results show that our model outperforms the state-of-the-art models for this task significantly. Ablation studies demonstrate that both Contrastive Pretraining and Node Clustering modules can significantly improve the performance of CSKG completion. Further experiments verify that our model can consistently improve as the sparsity of knowledge graphs increases: the higher the sparsity, the more improvement our model achieves.

heading: 2 Related Work
text: We briefly review traditional knowledge graph completion, and then pay more attention to commonsense knowledge graph completion.
Traditional Knowledge Graph Completion Many knowledge graph completion methods have been proposed, which can be classified into three types: embedding-based completion methods, pathfinding-based completion methods, and logical rule-based completion methods. The embedding-
based method learned the relation and node embedding by shortening the distance between the head-relation pair representation and the tail node representation, which had good scalability. Moreover, convolution has been proven to be an effective operation for acquiring the head-relation representation (Dettmers et al., 2018; Shang et al., 2019) in the embedding-based method. The path-based methods used the random walk inference based algorithm to find the related path. They achieved the prediction of missing tuples by comparing the related path with the relation to be predicted (Lao and Cohen, 2010; Khot et al., 2011). The rule-based methods utilized the induction rules to simplify the path-finding process in the knowledge graphs (Ren et al., 2020; Yang and Song, 2020). We refer the reader to (Ji et al., 2022) for more details about knowledge graph completion.
Commonsense Knowledge Graph Completion The existing methods assume specific relations, dense edges, and sufficient training samples in the knowledge graph. However, the sparsity of edges and the abstract nature of relations pose challenges for directly applying these methods to the CSKG.
Early methods in this field employed a strategy where, for a given head and tail, all relations were used to generate a large number of tuples. BiLSTMs and linear transformations were then utilized to score these tuples, allowing for the prediction of missing tuples in the CSKG (Saito et al., 2018; Li et al., 2016; Jastrzebski et al., 2018). Building upon this foundation, Shen et al. (2022) took a step further. They normalized the tail nodes and used RoBERTa to score these tuples. This approach resulted in the curation of Dense-ATOMIC, a CSKG with increased coverage and a richer set of multihop paths.
these methods did not take into account structural information and may suffer from computational inefficiency during inference. Another line of studies attempted to use the translated-based approach to reduce computational consumption (Malaviya et al., 2020; Ju et al., 2022; Wang et al., 2021). Furthermore, they incorporated GCN for extracting graph representations and fine-tuned BERT in the CSKG to obtain semantic representations. This integration capitalizes on the complementary nature of these two representations. In addition, these methods have applied convolutional layers to enhance the performance of CSKG completion. This further improves the effectiveness of the com-
pletion task. However, it was observed that simply fine-tuning BERT (Devlin et al., 2019) with pretraining tasks is hard to effectively capture the sentence-level semantic connection between two nodes. In response to this limitation, Su et al. (2022) proposed MICO, which learns distinct node representations for head nodes under varying relations, proving evidence that high-quality node representation can significantly aid CSKG completion. Moreover, different nodes in the CSKG may express the same concept. The node redundancy will bring difficulties to the commonsense inference on the CSKG (Jung et al., 2022), while the previous CSKG completion methods ignored.
Compared to these methods, we address the issue of node redundancy by employing a clustering algorithm. Our work stands out by learning the semantic representations of nodes through sentencelevel semantic connections and integrating latent concept information into node representations.

heading: 3 Task Definition
text: Given a CSKG G = (N,V,R) where N is the set of nodes, V is the set of edges and R is the set of relations. We regard each tuple vi = (h, rel, t) in the CSKG as a sample, which is composed of head node h, tail node t and relation rel, where h, t ∈ N , vi ∈ V , and rel ∈ R. Given a query (h, rel) formed by a head node h and a relation rel, the target of the CSKG completion is to maximize the score of the tail node t. Following the
previous work (Malaviya et al., 2020), for an edge (h, rel, t) existing in the CSKG, we also add an inverse edge (t, rel−1, h) to the graph, where rel−1 is the inverse relation of rel.

heading: 4 Approach
text: In this paper, we propose a new framework, CPNC, for CSKG completion using an Encoder-Decoder architecture (see Figure 2). The Encoder incorporates semantic, graph structure, and latent concept representation obtained from Contrastive Pretraining, Graph Convolutional Network (GCN), and Node Clustering, respectively, to acquire node representations. Given query (h, rel), the Decoder ranks the nodes in the candidate set N and finds tail nodes for the query by the rank.

heading: 4.1 Contrastive Pretraining
text: The current mainstream approach for completing CSKG utilizes translation-based methods and relies on GCN to represent the graph structure of nodes. Although GCNs are effective in modeling graph structure, they have limitations in capturing graph structure information in CSKG due to sparse edges. To address this issue, these methods incorporate semantic information by fine-tuning BERT on CSKG using Masked Language Model (MLM) and Next Sentence Prediction (NSP) tasks.
However, those methods are imperfect in modeling CSKGs composed of nodes presented as short text. MLM, which is a token-level task, focuses on
modeling the connection between masked words and other words, making it unsuitable for achieving sentence-level node representation. Similarly, NSP, while capable of modeling the semantic connection between two nodes at the sentence level, suffers from a mismatch between the input of the pretraining phase and the CSKG completion phase. During pretraining, NSP requires a pair of head and tail nodes for prediction, but during CSKG completion, only a single node is used for sentence representation. Consequently, NSP is not well-suited for the task of learning node representations, and similar observations have been made in the field of sentence representation learning (Li et al., 2020).
To obtain better node representations for the CSKG completion, we introduce Contrastive Pretraining (CP), a new method that sufficiently leverages semantic information at the sentence level. CP focuses on CSKG completion and differs from MICO (Su et al., 2022) by not incorporating relation categories. Instead, it fine-tunes BERT’s node representation using contrastive learning and capture sentence-level connections between nodes.

heading: 4.1.1 Building Contrastive Learning Samples
text: Assuming node pairs linked in the CSKG are semantically related, we consider node pairs without a link in the CSKG as semantically unrelated.
For each arbitrary edge (h, rel, t) in CSKG, we randomly sample a node t̄ having no linking with the head node h as its hard negative tail and construct a contrastive learning training sample c = (h, t, t̄). As shown in Figure 3, “take breakfast” and “cook food” are the head and tail nodes in a edge of CSKG. We randomly select a node “find information” having no linking with “take breakfast” to construct a training sample (“take breakfast”, “cook food”, “find information”). The previous methods primarily focused on the
head nodes while constructing negative samples, neglecting the importance of capturing unrelated semantics among the tail node set. To tackle this problem, we additionally constructed a contrastive learning sample creverse = (t, h, h̄) for the tail node t in the edge (h, rel, t).

heading: 4.1.2 Multiple Negatives Ranking Loss
text: We employ BERT for node embedding by inputting all nodes in a batch separately and applying Mean Pooling on the final layer’s output. Then, we use x(n) to denote the representation of a specific node n obtained through BERT.
In order to ensure that semantically related nodes are positioned closely in the embedding space, we utilize contrastive learning and employ the Multiple Negatives Ranking Loss for contrastive learning.
For a given sample ci = (hi, ti, t̄i) in a batch, we employ the Multiple Negatives Ranking Loss. We consider hi paired with ti as the positive sample while treating hi paired with other tail nodes tj in the batch as negative samples. Additionally, we consider hi paired with all hard negative tails in the batch as negative samples as well. The objective is to minimize the semantic distance between the nodes in positive samples within the batch. The formulation of the multiple negatives ranking loss is as follows:
L = − M∑
i=1
log eD(hi,ti))
∑M j=1 e D(hi,tj) + eD(hi,t̄j) , (1)
where D(h, t) = (x(h) · x(t)T )/(∥x(h)∥ · ∥x(t)∥) represents the cosine similarity of two nodes, M is the number of samples in the a batch.

heading: 4.2 Encoder
text: The Encoder aims to generate node representations for CSKG completion,as illustrated in the left part of Figure 2.
The node representation comprises two components,
1) the semantic representation matrix of nodes Esem obtained through Contrastive Pretraining:
Esem = Contrastive-Pretraining(N), (2)
2) the graph structure representation matrix of nodes acquired using a Graph Convolutional Network (GCN),
Egraph = GCN(G). (3)
Specifically, esem and egraph represent the semantic and graph representations of a particular node, respectively, which can be derived from Esem and Egraph.

heading: 4.3 Node Clustering
text: On the basis of the semantic node representation esem, we use the K-means algorithm to cluster all nodes N in the graph as follows:
{C1, C2, ..., CK} = K-means(N), (4)
where Ci is the i-th cluster, and K represents the number of clusters. Nodes in Ci are considered to have the same latent concept. As shown in the Node Clustering part of Figure 2, “eat breakfast ”, “have breakfast ” and “take breakfast ” are in the same cluster and have the same latent concept.
We define the representation of the latent concept as the mean of semantic representation of all nodes in a cluster, denoted as mean(Ck), where Ck contains a set of nodes in the same cluster.
To acquire the final node representation, we combine the node semantic representation, esem, the graph representation, egraph, and the latent concept representation, mean(Ck). This fusion is performed as follows:
e = [esem; egraph;mean(Ck)] ·Wembedding, (5)
where Wembedding is a weight matrix. The resulting node embeddings are considered as the node embedding space.
Intuitively, Node Clustering enhances CSKG completion by leveraging information from nodes within the same latent concept.
We also use a progressive masking process, following Malaviya et al. (2020), to integrate latent concept information. Initially, the latent concept representation is fully masked, and throughout the first 100 training epochs, it is gradually unmasked, improving the overall performance.

heading: 4.4 Decoder
text: The goal of Decoder is to rank the nodes in the candidate set, which is considered as N in our work.
In order to obtain the query representation, we first use a convolutional layer to fuse the relation and head node representation:
q = Conv(eh, erel), (6)
where eh is the representation of the head nodes in node embedding space obtained from Eq. 5, erel is the representation of relation rel and Conv is a convolution operation.
Then, we predict a 0-1 distribution vector, based on the representation of the query q, indicating the likelihood of each node in the candidate set N becoming the tail node:
pt = δ(qWconvEN ), (7)
where Wconv is a weight matrix, EN is the representation matrix of candidate nodes in node embedding space obtained from Eq. 5, and δ is a sigmoid function.

heading: 5 Experiments

heading: 5.1 Experimental Setup

heading: 5.1.1 Datasets
text: We evaluate our CPNC framework on two CSKG completion benchmarks, i.e., CN-100K (Speer et al., 2017) and ATOMIC (Sap et al., 2019).
CN-100K is a dataset that encompasses general commonsense knowledge. This version contains 36 relation types and the Open Mind Common Sense (OMCS) entries from ConceptNet (Speer and Havasi, 2013). The average length of the nodes in CN-100K is 2.85 words. Following Malaviya et al. (2020)’s split, the training set contains 10,000 tuples, and the validation set and test set both contain 1200 tuples.
ATOMIC is an atlas of everyday commonsense reasoning and primarily focuses on event-level commonsense knowledge in the form of if-then relations. It comprises 9 relation types, with an average of 4.40 words per node. We split ATOMIC following Malaviya et al. (2020)’s work, where the training set consists of 610,536 tuples, while the validation and test sets contain 87,700 tuples and 87,701 tuples, respectively.

heading: 5.1.2 Evaluation Metric
text: We evaluate the performance of our method using MRR and HITS, following previous CSKG completion methods (Wang et al., 2021; Malaviya et al., 2020; Ju et al., 2022). The results are reported by averaging over both forward tuples (h, rel, t) and inverse tuples (t, rel−1, h). Moreover, because the nodes in CSKG are represented in free-form text, it is possible for nodes other than the golden tail nodes to be considered reasonable tail nodes. To address this, we conduct a human evaluation to assess the predictions made by our models.

heading: 5.1.3 Implementation Details
text: To perform Contrastive Pretraining, we use the contrastive learning framework provided in https:// github.com/UKPLab/sentence-transformers. Our approach employs BERT-large as the base model, which contains 340M parameters. During training, we utilized a batch size of 128 and conducted training for 3 epochs on the CN-100K and ATOMIC datasets. We choose the Adam optimizer for optimization, setting the learning rate to 1e-4 for BERT-large and 5e-5 for the MLP. For the remaining hyperparameters, we used the default values provided by the framework.
For the CPNC-I and CPNC-S model, we use experimental settings proposed by Malaviya et al. (2020) and Wang et al. (2021), respectively. Both models are trained for a minimum of 200 epochs using BERT-large, which consists of 340M parameters, to encode semantic representations. During training, we evaluate the MRR on the development set every 10 epochs for CN-100K and ATOMIC. Training continues until no further improvement in
MRR is observed. We select the model checkpoint that achieves the highest MRR on the development set for testing.

heading: 5.2 Compared Systems
text: We compare our approach with nine baseline systems across three categories.

heading: 5.2.1 KG-adapted Methods
text: We adapt classic knowledge graph completion methods for CSKG completion. DistMult (Yang et al., 2015) employed a bi-linear product to calculate score of a tuple; ComplE (Trouillon et al., 2016) utilized imaginary number representation to effectively handle a large number of relations in the knowledge graph; ConvE (Dettmers et al., 2018) fused the representation of the source node and the relation through a 2D convolution layer to obtain the representation of query (h, rel); ConvTransE extended ConvE by incorporating the translational properties of TransE.

heading: 5.2.2 Generation-based Methods
text: COMeT (Bosselut et al., 2019) is a Transformerbased knowledge generation model. Following Malaviya et al. (2020), we adapt COMeT for the CSKG completion and only evaluate in the forward direction. Additionally, we also use their modifications to the ranking method used in COMeT. Specifically, COMET-Normalized and COMETTotal use the normalized negative log-likelihood scores and total log-likelihood scores, respectively, to rank nodes in the candidate set.

heading: 5.2.3 CSKG-dedicated Methods
text: SIM+GCN+BERT+ConvTransE (SGBC) densified the CSKG by connecting the synthetic edges between similar semantic nodes, improving the graph structure representation (Malaviya et al., 2020). Moreover, they used fine-tuned BERT to encode the semantic information of the nodes.
InductivE (Wang et al., 2021) is proposed to enhance the unseen entity representation with neighboring structural information by densifying Graph. Relational graph attention networks (RGAT) are proposed weighted the importance of neighbor nodes of each node to obtain a better node representation.

heading: 5.2.4 Our CPNC Methods
text: By incorporating CPNC with two CSKG-dedicated methods (SGBC and InductivE), we obtain two models, i.e., CPNC-S and CPNC-I.

heading: 5.3 Main Results
text: In Table 1, we report experimental results of KGadapted methods, Generation methods, CSKGdedicated methods and our methods on CN-100K and ATOMIC.
On CN-100K, both KG-adapted methods and Generation methods exhibit unsatisfactory performance, with MRR values below 21% and HITS@1 values below 14%. These results demonstrate that directly applying KG-adapted and Generation methods to CSKG completion is ineffective. In contrast, CSKG-dedicated methods achieve significantly better results on CN-100K, with MRR values above 43% and HITS@1 values above 30%. These metrics are twice as high as those obtained by KG-adapted and Generation methods, highlighting the advantage of CSKG-dedicated methods. Comparing our proposed CPNC-S and CPNC-I models with mainstream CSKG-dedicated methods (SGBC and InductivE), we observe performance improvements of 5.40% and 2.08% on MRR, respectively. Notably, our CPNC-I model sets a new state-of-theart result on CN-100K.
On ATOMIC, we draw a similar conclusion. The KG-adapted methods exhibit poor performance, while CSKG-dedicated methods achieve better results. In addition, compared with the SGBC and InductivE models, our CPNC-S and CPNC-I models outperform the SGBC and InductivE models by 2.89% and 1.19% on MRR, respectively. And the CPNC-I model achieves the state-of-the-art result on ATOMIC.
We conduct Paired t-Test and the result proves that the improvement of CPNC is significant.

heading: 5.4 Ablation Study
text: To demonstrate the effectiveness of Contrastive Pretraining (CP) and Node Clustering (NC) on CSKG completion, we perform the ablation study where we removed CP and NC. The results are presented in Table 2.
When NC was removed, we can observe that the performance of both CPNC-S and CPNC-I decreases on CN-100K and ATOMIC. This indicates that NC’s latent concept representation effectively assists CSKG completion. It is worth noting that CP can still bring improvement compared with the previous method, showing that the semantic representation provided by CP is crucial for CSKG completion.
CPNC, which combines NC and CP, consistently yields the highest result. By replacing CP with finetuned BERT (Malaviya et al., 2020), the performance of both CPNC-S and CPNC-I drops. However, they still achieve comparable results with the previous CSKG-dedicated methods. This demonstrates that the CP requires a high-quality semantic representation of nodes to acquire good latent concept representation.

heading: 5.5 Human Evaluation
text: For a given query (h, rel), additional nodes besides the golden tail node could also become reasonable tail nodes. For example, given a query “(do housework, Causes) ”, besides golden tail node “ clean the house”, “ house get clean” and “ clean the room” are also reasonable tail nodes. However, automated
metrics fail to cover these nodes. To address this issue, we randomly select 200 queries from the test set of the CN-100K and ATOMIC. For each query, we use our methods and current mainstream methods to rank the nodes in the candidate set, We manually calculate the number of reasonable tail nodes in the top 10 candidates. The results are shown in Table 3.
Observing human evaluation results, the CPNC model has an excellent performance in CSKG completion. Besides, our method is significantly higher than the current mainstream methods under the human evaluation results, which further validates the effectiveness of our approach.

heading: 5.6 Model Adaptability
text: The CPNC can be effectively applied to various methods. In our experiment, we incorporate CPNC into SGBC and InductivE, resulting in significant improvements.
The SGBC method initializes the input of GCN randomly and combines graph structural representation acquired by GCN and semantic representation obtained by BERT as the node representation; InductivE initializes the input of GCN with BERT and uses the output of GCN as the node representation. Despite the differences in their structures, both methods benefit from the application of CP and NC.
On CN-100K, CPNC-S and CPNC-I bring improvements of 5.40% and 2.08% in MRR, respectively; On ATOMIC, CPNC-S and CPNC-I yield MRR improvements of 2.89% and 1.19%, respectively. This result demonstrates the adaptability of our method.

heading: 5.7 Discussion on the Effect of Relieving Sparsity
text: To evaluate the effectiveness of CPNC in mitigating the impact of edge sparsity in the CSKG, we conducted a comparative study between CPNC-S and SGBC on CN-100K with varying sparsity levels. The removal of edges in the CSKG results in
increased sparsity, with a higher number of deleted edges corresponding to a higher sparsity degree.
As shown in Figure 4, when the sparsity degree increases, CSKG completion performance deteriorates. CPNC-S consistently outperforms SGBC in terms of MRR on CN-100K across different sparsity levels, indicating CPNC’s ability to mitigate edge sparsity in the CSKG. CPNC-S achieves a 12.48% and 3.95% increase in MRR compared to SGBC at sparsity extents of 25% and 50% respectively. These results demonstrate that CPNC is more effective in alleviating edge sparsity in the CSKG, with greater improvements observed for sparser graphs.

heading: 5.8 Discussion on the Number of Latent Concepts
text: Different numbers of clusters can lead to varying levels of granularity in latent concepts, which in turn can impact the performance of CSKG completion. In order to investigate this, we conducted experiments with the different numbers of clusters
on CN-100K and ATOMIC datasets and present the results in Figure 5.
On CN-100K, we observed an overall increasing trend in completion results as the number of clusters increased. However, the rate of improvement tends to diminish after reaching 900 clusters. The maximum MRR achieved when using the number of clusters is 1200.
On ATOMIC, the MRR initially increases with an increasing number of clusters but starts to decline once the number of clusters reaches 900.
Based on our main results, we selected 1200 clusters for CN-100K and 900 clusters for ATOMIC, allowing us to effectively capture latent concepts through cluster nodes.

heading: 5.9 Discussing on Contrastive Learning Loss
text: In order to obtain a better semantic representation of nodes, we experiment with five kinds of contrastive learning methods and observe their performance in CSKG completion. The result is shown in Table 4.
In order to compare those contrastive learning methods, we use the node embedding obtained by those methods to complete CSKG without node clustering. Each contrastive learning method yields distinct results, and our main result selects Multiple Negatives Ranking Loss due to its highest MRR.

heading: 6 Conclusion
text: In this work, we propose a new CSKG completion framework CPNC to address issues arising from node redundancy and edge sparsity. CPNC obtains better semantic node information through Contrastive Pretraining, which alleviates the problems caused by edge sparsity. CPNC also utilizes the latent concept representation acquired through Node Clustering to alleviate the problem caused by node redundancy. On CN-100K and ATOMIC, experimental results and extensive analysis demonstrate the effectiveness of Contrastive Pretraining and Node Clustering.
Limitations
Due to the limitation of time and resources, in this work, we select a relatively small number of clusters during the clustering process, which results in coarse-grained clustering. Fine-grained clustering can provide a better latent concept but will also lead to increased computational resources and time consumption. We will attempt to trade-off between the cost and the granularity of clustering in future work to further explore the impact of the latent concept on CSKG completion. Besides, our Node Clustering module is not integrated in an end-toend manner in our work; We will consider using the topic neural network to construct an end-to-end model.
Ethics Statement
We would like to thank Malaviya et al. (2020) and Wang et al. (2021) for their code on commonsense knowledge graph completion. Their models are licensed under MIT, which allows copying, modifying, merging, publishing, and distributing of the material.
In the stage of human evaluation, we employed three graduate students experienced in natural language processing for human evaluation. We paid the graduate students about $8 per hour, well above the local average wage, and engaged in constructive discussions if they had concerns about the process.

heading: Acknowledgements
text: This work was supported by the Natural Science Foundation of China (No. 62076133), and the Natural Science Foundation of Jiangsu Province for Distinguished Young Scholars (No. BK20200018).

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations section
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract section and section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5.1.3 Implementation Details

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 5.1.3 Implementation Details

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Ethics Statement Section

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 5.1.3 Implementation Details
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use publically available datasets, and the authors of the dataset have made the corresponding declaration.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We will provide those data after the review process.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 5.1.1 section Datasets
C 3 Did you run computational experiments?

heading: 5.1.3 section Implementation Details

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5.1.3 Implementation Details
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5.1.3 Implementation Details

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5.3 Main Results

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5.1.3 Implementation Details
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? We perform simple human annotation and evaluation, there is no need of providing the full text.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Ethics Statement section
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Ethics Statement section

heading: Future Work
text: We will attempt to trade-off between the cost and the granularity of clustering in future work to further explore the impact of the latent concept on CSKG completion. Besides, our Node Clustering module is not integrated in an end-toend manner in our work; We will consider using the topic neural network to construct an end-to-end model