Gold: A Global and Local-aware Denoising Framework for Commonsense Knowledge Graph Noise Detection
Authors: Zheye Deng
Weiqi Wang
Zhaowei Wang
Xin Liu
Yangqiu Song
Venue: EMNLP 2023 Findings (2023)
Keywords: Knowledege Graph Denoising
Commonsense Reasoning
Question Answering

Abstract: Commonsense Knowledge Graphs (CSKGs) are crucial for commonsense reasoning, yet constructing them through human annotations can be costly. As a result, various automatic methods have been proposed to construct CSKG with larger semantic coverage. However, these unsupervised approaches introduce spurious noise that can lower the quality of the resulting CSKG, which cannot be tackled easily by existing denoising algorithms due to the unique characteristics of nodes and structures in CSKGs. To address this issue, we propose Gold (Global and Local-aware Denoising), a denoising framework for CSKGs that incorporates entity semantic information, global rules, and local structural information from the CSKG. Experiment results demonstrate that Gold outperforms all baseline methods in noise detection tasks on synthetic noisy CSKG benchmarks. Furthermore, we show that denoising a real-world CSKG is effective and even benefits the downstream zero-shot commonsense question-answering task. Our code and data are publicly available at https://github.com/HKUST-KnowComp/GOLD.

GOLD: A Global and Local-aware Denoising Framework for Commonsense Knowledge Graph Noise Detection
Zheye Deng, Weiqi Wang, Zhaowei Wang, Xin Liu, Yangqiu Song Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China {zdengah, wwangbw, zwanggy, xliucr, yasong) ust.hk
Abstract
Rene Magritte
run out of paint
brush your feeth
Commonsense Knowledge Graphs (CSKGs) crucial for commonsense reasoning, yet constructing them through human annotations can be costly. As a result, various auto- matic methods have been proposed to con- struct CSKG with larger semantic coverage. However, these unsupervised approaches intro- duce spurious noise that can lower the qual- ity of the resulting CSKG, which cannot be tackled easily by existing denoising algorithms due to the unique characteristics of nodes and structures CSKGs. To address this issue, propose GOLD (Global and Local-aware Denoising), denoising framework for CSKGs that incorporates entity semantic information, global rules, and local structural information from the CSKG. Experiment results demon- that GOLD outperforms all baseline meth- ods in noise detection tasks synthetic noisy CSKG benchmarks. Furthermore. we show that denoising real-world CSKG is effective and even benefits the downstream zero-shot commonsense question- question-answering answering task. Our code and data publicly available https: Seggiuuttcoo//KKKSSSSTToooCCCCCO Knw.Con//OOLDCDD
A4
NotDesirex
EverFFor
Capabletf
painter
paint
Capubletty
CapableOy
HesPreeepiinn
UxestFor
point door
point house
brush
Figure A subgraph CSKG with two instances of noise. One noise is (paint, UsedFor, brush your teeth), as there should be no relation between these two nodes. Another noise is (painter, ISA, paint house) because the correct relation should be CapableOf.
proposed to acquire commonsense knowledge at scale (Bosselut et al., 2019), including prompting Large Language Model (LLM) (West et al., 2022; Yu al., 2022), rule mining from massive cor- pora (Tandon et al., 2017: Zhang et al., 2022a), and knowledge graph population (Fang et al., 2021 a,,,a 2023). Although those methods are effective, they still suffer from noises introduced by construction bias and the lack of human supervision. Therefore. how to identify noise in large-scale CSKG accu- rately and efficiently becomes crucial research question.
1 Introduction
The emergence of Commonsense Knowledge Graphs (CSKGs) has significantly impacted the field of commonsense reasoning (Liu et al., 2021; Zhang et al., 2020) as CSKGs provide common- sense knowledge that often not explicitly stated in the text and difficult for machines to capture systematically (Davis and Marcus, 2015). While existing methods bank on expensive and time- consuming crowdsourcing to collect commonsense knowledge (Sap al., 2019a; Mostafazadch et al., 2020), remains infeasible to obtain CSKGs that are large enough cover numerous entities and situations the world (He et al., 2022; Tandon et al., 2014). To overcome this limitation, various automatic CSKG construction methods have been
To tackle this issue, noise detection algo- rithms have been proposed for conventional entity- based KGs by primarily adopting two approaches: Deaming-bassed and rule-based. Learning-baat based methods like TransE (Bordes et al., 2013) learn representations entities and relations that adhere to specific relation compositions like translation assumption or relational rotation. To enhance their performance, researchers also incorporate local in- formation around the head and tail entities, such different paths from head to tail (Lin al., 2015; Xie et al., 2018; Jia et al., 2019) and neighbor- ing triples (Zhang et al., 2022b). These methods aim to improve their ability to capture the complex relationships between entities in KGs. However,

---

they are not easily adaptable to the unique char- acteristics of CSKGs. In CSKGs, nodes are non- canonicalized, free-form text, meaning nodes with different descriptions may have related semantics. As illustrated in Figure 1, "paint door" and "paint house" are two distinct nodes but imply related se- mantics (Speer et al., 2017). Additionally, when detecting noise (paint, UsedFor, brush your teeth), "brush your teeth" is isolated node that cannot be distinguished based on any structural information. Only through the power of a language model can it be learned that "paint" and "brush your teeth" are uncorrelated, thus detecting such noise. The aforementioned methods overlook this semantic information and cannot generalize to semantically similar events with diverse structural information.
rule encoder to generalize the learned rules and guide noise detection. The local detector, inspired by Zhang et al. (2022b), adopts graph neural net- work to efficiently measure the similarity of aggre- gated semantic information of neighboring triples of the head and tail nodes to help detect noise. Ex- tensive experiments on two manually synthesized noisy-CSKG benchmarks demonstrate the efficacy and state-of-the-art performance of GOLD. Further experiments and analyses with ATOMIC (West et al., 2022), large-scale CSKG distilled from GPT3, demonstrates its proficiency in identifying noise within real-world CSKGs, while also yielding advantages in the downstream zero-shot common- sense question-answent In summary, in this paper, we make the following contributions:
task.
On the other hand, rule-based methods utilize logical rules in KGs for noise detection. For in- stance, as shown in Figure the correct relation between "painter" and "paint house" should be CapableOf. This can be easily detected through the learned logical rule: (ff y) <- Belth
We introduce a new task: CSKG denoising. which can be applied to various CSKG con- struction and LLM distillation works.
We propose a novel framework GOLD, which outperforms all existing methods (Section 6.1) and LLMs (Section 6.3).
Capable0f(z; z) (a. Hastarerreuuiiitee//eee
et al. (2020) similarly propose an approach based on information theory that extracts sub-graph pat- terns to identify the noise. However, the sparsity of edges in CSKGs (Malaviya al., 2020) posits serious challenge learning structural information well, as the number of learnable rules decreases significantly. This requires generalizable rule- learning ability at the noise detector side to expand the rule bank accordingly, which currently lack- ing. Therefore, applying noise detection models for KGs directly to CSKGs can result in incomplete learning of both semantic and structural informa- tion in the CSKGs
We show that GOLD successfully detects noises in real-world CSKGs (Section 6.5) and such denoising extrinsically benefits down- stream zero-shot commonsense question- answering task (Section 6.4).
2 Related Work
2.1 Knowledge Graph Noise Detection
Many existing knowledge graph noise detection approaches utilize some local information while simultaneously training embeddings to satisfy the relational assumption. Path information is the most commonly used type of local information, as the reachable path from the head entity to the tail en- tity has been proven crucial for noise detection in knowledge graphs (Lin et al., 2015; Xie et al., 2018; Jia et al., 2019). Zhang et al. (2022b) show that contrastive learning between the information of neighboring triples of the head and tail entities is more effective because of the triple-level con- trasting instead of entity or graph-level leading to superior performance compared to all path-based methods. Clustering methods (Ge et al., 2020) are also used to partition noise from triples, and an active learning-band classification model is pro- posed detect and repair dirty data. While these methods consider local information, our work also accounts for semantic information and the global
In order to detect noises in CSKGs effectively, it is important to consider both the semantic infor- mation and the global and local structural informa- tion jointly. However, these factors have not been given enough importance in existing denoising ap- proaches. To address this gap, we propose GOLD (Global and Local-aware Denoising), a CSKG noise detector that uses PLM-based triple encoder and two noise detectors that take into account both global and local structures (Section 4). Specifically, the triple encoder extracts the semantic informa- tion contained in the free-text formatted nodes in CSKGs. To identify correct patterns, the global detector uses high-frequency patterns extracted through rule mining, which intrinsically uses a

---

information of the knowledge graph to guide noise detection, better mitigating the impact of noise on local information. Regarding direct noise detec- tion in CSKGs, Romero and Razniewski (2023) study the problem of mapping the open KB into the structured schema of an existing one, while our methods only use the CSKG to be denoised itself, not relying on any other CSKG.
terns based on subgraphs help refine knowledge graphs, but requires type information of node and hence cannot be applied to the CSKG.
2.3 Knowledge Graph Completion with Pretrained Language Models
Aside from specifically designed noise-detection methods, the line of works targetting KG com- pletion can also be transferred to tackle noise- detection tasks. Previous research has shown that PLMs can achieve outstanding performance KG completion tasks for both conventional KGs (Wang and Li, 2016; An et al., 2018; Yao et al., 2019; Wang et al., 2021b; Markowitz et al., 2022; Shen et al., 2022) and CSKGs (Su et al., 2022; Yasunaga et al., 2022) due to their ability to capture lin- guistic patterns and semantic information. How- ever, two limitations still exist. First, performing edge classification using a PLM requires optimiz- ing large number of parameters on textual data that has been transformed from edges in CSKGs. Such fine-tuning not only computationally ex- pensive but also incapable of learning structural features in graphs, which are essential for accu- rately identifying and classifying edges. Second, recent studies (Safavi et al., 2021; Chen et al., 2023) have shown that language models, regardless of their scale, struggle to acquire implicit negative knowledge through costly language modeling. This makes them potentially vulnerable to noise detec- tion tasks, as these noises typically belong to nega- tive knowledge. Therefore, more sophisticated ma- nipulations of the semantic information extracted by PLMs are needed to leverage them for noise detection tasks efficiently.
2.2 Knowledge Graph Rule Mining
Another related line of work is knowledge graph rule mining. which is essential to our method. This task has received great attention in the knowledge graph completion. The first category of methods is Inductive Logical Programming (ILP) (Muggleton and Raedt, 1994), which uses inductive and log- ical reasoning to learn rules. On the other hand, AMIE (Galárraga et al., 2013) proposes a method of association rule mining, which explores fre- quently occurring patterns in the knowledge graph to extract rules and counts the number of instances supporting the discovered rules and their confi- dence scores. AMIE+ (Galáragaa et al., 2015) and AMIE 3 (Lajus et al., 2020) further improve upon this method by introducing several pruning opti- mizations, allowing them to scale well to large knowledge graphs. SWARM (Barati et al., 2017) also introduces statistical method for rule min- ing in large-scale knowledge graphs that focuses on both instance-level and schema-level patterns. However, it requires type information of entities, which is not available in the CSKG and, there- fore, cannot be applied to CSKG. Recently, with the success of deep learning, the idea of ILP has been neuralized, resulting in a series of neural- symbolic methods. Neural LP (Yang et al., 2017) and DRUM (Sadeghian al., 2019) both propose end-to-end differentiable models for learning first- order logical rules for knowledge graph reasoning. Despite the great success achieved by the combina- tion of Recurrent Neural Network (RNN) (Schus- ter and Paliwal, 1997) with rule mining (Qu et al., 2021: Cheng et al., 2022, 2023), neuralized meth- ods are intuitively hard to interpret due to the con- fidence scores output by neural networks. Further- more, jointly learning rules and embedding has been proven be effective (Guo al., 2016), and iteratively learning between them can also promote the effectiveness of both (Guo al., 2018: 8: Zhang et al., 2019b). For noise detection in knowledge graphs, Belth et al. (2020) learn higher-order pat-
3 Problem Definition
Noises in CSKG resents not only basic facts in traditional knowl- edge graphs but also the understanding possessed by most people (Liu and Singh, 2004), we evaluate whether triple is a noise from two perspectives:
Commonsense knowledge rep-
Truthfulness: It should be consistent with ob- jective facts. For example, (London, IsA, city in France) is not true because London is not in France but in England.
Reasonability: It should align with logi- cal reasoning and be consistent with cul- tural norms. For example, (read newspaper, NotivatedByGoal, want to eat vegetables) is not logically reasonable. The two nodes are

---

not directly related, and there no clear rela- tionship between them. Another example is that (hippo, AtLocation, kitchen) violates our understanding and experience of reality because hippos are large mammals that are highly unlikely and unrealistic to be found in kitchen.
specific canonicalized form. If their semantic in- formation is ignored, will inevitably affect the accuracy of noise detection. Therefore, the Triple Encoder (TE) employs a PLM to encode the se- manties of each node and relation. For instance,
considering an example of triple (h, r, t), their em- beddings are defined as:
If triple fails to satisfy any of the aspects men- tioned above, we define it noise.
LM(M), s, LM(r), LM()),
(I)
where LM is frozen PLM that maps the input text an embedding, To strike balance between capturing the relationship between h, r, and and maintaining model efficiency, we opt an efficient RNN as our encoding method for the CSKG triples:
CSKG Denoising A CSKG can be represented G (V. E,),, where V is set of nodes, R is set of relations, and & C V x R x V is set of triples or edges. Given triple (h, E & in CSKG, we concatenate the language descriptions h, r. and i and determine whether this descrip- tion conforms to commonsense. We note that each triple violates commonsense to a different degree, and we define noise detection as ranking prob- lem to standardize the evaluation process better. ranking pro-
Ch. cr, (,,.
Then, we simply concatenate them together to get the representation of the triple (h, r)::
TE(h, r, [ee]]e.[led)
(3)
Thus, model noise detection cess where a scoring function ʃ & -> IR indicates the likelihood of the triple being noisy.
4.2 Global Rule Mining
4 The GOLD Method
To detect noisy triples, scoring (h, r, t) only from local perspective, such as modeling the neighbors of h and t, or analyzing the path from h to t may not be sufficient eliminate the interference of noisy triples, as is difficult to determine what is noise from local structures alone. commonsense knowledge graphs, the noise ratio should not be excessively high. So, learning high-frequency pat- terns from global perspective is likely to cover correct triples. In turn, patterns can guide us in identifying the noise data when detecting viola- tions.
Our proposed method GOLD comprises four com- ponents: triple encoder, global noise detector, lo- cal noise detector, and comprehensive evaluation scorer. An overview is presented in Figure 2. First, we leverage a PLM to encode the natural language descriptions of nodes and relations in CSKGs to obtain their sentence embeddings, thus further en- coding the triples. When detecting noise, we evalu- ate the likelihood of a triple being noise from both a global and local perspective. From the global perspective, we aim to identify high-frequency pat- terns in the knowledge graph, as small amount of noise is less likely to affect correct high-frequency patterns (Belth et al.. 2020). To accomplish this, we employ rule mining to extract high-quality rules from the knowledge graph. From the local per- spective, we adopt graph networks to aggregate the neighboring triple information around both the head and tail nodes of given edge, allowing us to estimate if there is any correlation. Finally, based on these two aspects of detection, we obtain com- prehensive score indicating the noise level.
To incorporate the global information of the en- tire CSKG when determining the probability of triple being noise, we use the method of rule min- ing to first extract high-frequency, high-confidence, and interpretable rules from the CSKG. Taking into account both the interpretability and efficiency of the model, we employ AMIE 3 (Lajus et al., 2020), rule mining method based on the frequency of each pattern, to generate logical rules automatically with the following format:
[x v) +- rhy (1) A Thy (26-1,1118))
(4)
4.1 Triple Encoder
where Th (x y) is rule head and rby (I. 21) A..
As we mentioned earlier, the nodes in CSKG are linguistic descriptions that are not restricted to any
The (2 y) is rule body, H, y. 21, 2k-1 are nodes, The Tb The are relations. As depicted in

---

Triple Frender
Triple Eneoder
PLM
PLM
PLM
Tmple
Thple
Cameelletiy
cook
hunury
have
food
(1) Triple Encoder
(2) Global Rule Mining
eat meal
rr..
to store
r,) I E (h,
EledFirr
kitchen
FoeeFir
book meal
Madhindiahreey
buy food
CundPar
prejare mecl
CaeenBeii
Conse
HasFirsSiohevnt
(h,
hungry
have lunch
CasseiAasiee
Local Neighbor Triple Learning
(4) Jointly Estimation
Overview of our GOLD framework. The example showing how to examine the noise level of the triple (cook meal, HasFir iastbbett,ee buy food) from both global and local perspectives are presented in the figure. The rule hasFirstSubevent.e y) CausesDesire which is learned from the entire CSKG provides guidance to noise detection, while the neighboring triples of "cook meal" and "buy food" are used for aggregation as features for local structure learning.
Figure 2:
y) CausesDesire 22) HHasFirstSubevent
Equation (4), the rule body consists of triples:
4.3 Local Neigboring Triple Learning
f = (x, 21, b2, 22). ts (zt-1. be, v).
Structural information plays significant role in en- hancing performance for KG noise detection tasks. Most methods require that the relationship between two nodes should be equivalent to translation be- tween their embeddings (Xie al., 2018: Zhang et al., 2022b). We relax this restriction and aim to determine some level of contextual correlation between two related nodes. As for the specific relation, our global rule mining component will learn its corresponding representation. To capture the contextual semantic information of the triples around nodes, we adopt Graph Attention Network (GAT) (Velickovic al., 2018) to aggregate the information of the neighboring triples.
(5)
To address the issue of poor generalization of
mined rules due to sparsity in edges in CSKGs, we consider rule body
sequence and em- ploy RNN the neuralized Rule Encoder (RE)
generalize the rules:
RE(rs) RNN (TE())) TE(t2). ... TE(++))
(6)
Specifically, for each relation as the rule head, we retain the top krules rules with the highest confi- dence score given by AMIE 3 for training the rule encoder. In cases where there is no correspond- ing instance for rule body, we fill all triples in the rule body with (z, h. y) to align the energy scores of the other triples. And believe that well-generalized rule encoder can learn repre- sentation that can explicitly infer the rule head rh- i.c., (z, h, y). Hence, we align the dimensions of the outputs from TE and RE and define the energy function as follows:
We use a transformation matrix W E Fxd to map the i-th triple (h, ri, f,) to the embedding
v. W [le,
where F is the dimension of the latent space and is the embedding dimension of the triple, and perform the self-attention function R R R on the triples to get Wij = (v;, 02)) which indicates the context of the j-th triple to the i-th triple. To compute the attention of the neighboring
Egloball (h, r,t)=
Σ (Pr.P))
RE(r))) TE(rx)ll-
(7)

---

triples on the head and tail nodes, respectively, we define the neighboring triples of the node e as N₁. [(h, F, 0)ı = e evi = e}. and then use the softmax function to normalize the coefficients:
ative examples & is constructed based on it by
replacing either h t with a random node V/:
& ((ē,r,t) v) u (, e v)
(14)
Experimental Setup
agyth: softmax (W(yt))
exp(w10()
5.1 Datasets
ENWEE exp(wxem)
To evaluate the detection capability of denoising models, we follow the method introduced by Xie et al. (2018) construct benchmark datasets for evaluation, which involves generating noise with manually defined sampling rules and injecting it back into the original CSKG. We select Concept- Net (Speer al., 2017) and ATOMIC (Sap et al., 201 2019a) as two source CSKGs due to their man- ageable scale and diverse coverage of edge seman- tics, including various entities, events, and com- monsense relations. Since these manually curated CSKGs do not contain noise naturally, we synthe- size noise for each CSKG separately using meticu- lously designed rules, as done by Jía al. (2019). that incorporate modifications on existing edges and random negative sampling. This approach, as demonstrated by Jia et al. (2019), ensures that the resulting noises not only maintain being highly in- formative, thus more challenging for the model to detect, but also stimulate several types of noise that may appear in real- world CSKGs. More details for noise synthesis provided in Appendix A.I.
9g0 = softmax,(0) (wn)(o)
exp(W,s())
EmmEN, exp(m
where aijon) represents the attention of the j(k)_ th triple on node h4. while Biytes represents the attention of the j(t)-th triple on node 4, It is worth noting that the -(h) triple is required to meet the condition of being neighbor of node hú, and similarly, the ,(h)-th triple must also be neighbor of node 44-
We use the normalized attention coefficients to calculate a linear combination of the corresponding embeddings, which then serves as the final output:
P. Eggheen, (gono,as)
(10)
(E, [giee,, Vy
9: where P, is obtained from the perspective of the neighbors of node h4 q; is obtained from the per- spective of the neighbors of node t, and a repre- sents nonlinearity.
5.2 Evaluation Metrics
We simply employ the Euclidean distance be- tween them to measure the correlation between h; and & and obtain the energy function of triple (hi, ris dú) under local perception follows:
We use two common metrics to evaluate the perfor- mance of all methods.
Recall @k. Given that there are k noisy triples in the dataset, we sort all triples by their score descending order, where higher score indicates a higher probability of being a noisy triple. We then select the top & triples and calculate the recall rate:
Ejocal r..4)) 4)) = |pp - 9,112.
(11)
4.4 Jointly Learning and Optimization
The overall energy function of each triple (h, r. obtained by combining the global and local energy functions. We have:
Recall@ k Noisy Triples in the top-k list
(15)
Area Under the ROC Curve (AUC) mea- sures the probability that a model will assign a higher score to a randomly chosen noisy triple than randomly chosen positive triple. A higher AUC score indicates a better performance.
AUC.
Ahh, r, t) Fglobal r,,) + (h, (12)
where À is hyperparameter.
We use negative sampling to minimize the margin-based ranking loss
L= Σ Σ max StEE
E)i (13)
5.3 Competing Methods
We compare our model with state-of-the-an mod- els, which can be mainly divided into three cate- gories: (i) structure embedding- based methods that are unaware of noise, including TransE (Bordes
where i! represents positive triple (h, r, t), and i represents a negative triple. We follow the set- ting of DistMult (Yang et al., 2015); a set of neg-

---

ConceptNet
ATOMIC
Model
N5
N10
N20
N10
N20
R@55
AUC
R
10
AUC
R@20
AUC
R@5
AUC
R 10
AUC
R 20
AUC
TransE
.84
.679
.163
.670
.276
.665
.390
849
.475
.849
.569
.853
DistMult
.118
.656
.187
.652
.283
.653
.255
.841
.490
.835
.551
.840
ComplEx
.660
.733
.248
.720
.364
.718
.600
.842
.531
.841
.581
.839
RotatE
.114
.614
.177
.609
.262
.604
.140
.738
.212
.732
.311
.728
CKRL
.150
.693
.211
.701
.694
.317
.787
.411
.794
CAGED
.474
.903
.536
.620
.877
.577
.914
.630
.910
.674
KG-BERT
.601
.925
.680
.936
.750
.939
.714
.936
.782
.953
.813
.951
LASS (BERT-base)
.640
.955
.706
.955
.768
.951
.762
.956
.791
.956
.821
.955
LASS (BERTlarge)
.689
.959
.750
.963
.804
.961
.757
.957
.792
.957
.827
.957
LASS (RoRERTa-hase)
.665
.611
.709
.775
.955
.775
.961
.802
.960
.831
.959
LASS (RoBEKIa-large)
.30
.785
.91
.780
.964
.814
964
.844
GOLDD (RoBFKTa-hsee)
831
982
.847
.800
.866
.974
.861
.964
.880
.65
.887
.958
GOLD (RoBERTa-large
.288
.885
.841
.978
.868
.977
.864
.968
.880
.962
.900
.968
GOLD (DeBFKTb-vi-hase)
8399
9979
.861
980
.875
.975
.862
.965
.873
.967
.884
.959
GOLD (DeBBKlb-<3-arge))
823
9973
.850
9975
863
.668
8849
.962
.863
.988
.880
.61
GOLD (Sentence- 15-base)
838
.983
.852
.981
.870
.975
.863
.959
.890
.964
.896
.988
GOLD
822
.982
8336
.979
858
.973
.862
.960
.880
.962
.891
.600
GOLD (Sentence
.842
.985
.859
.981
.878
.979
.872
.969
.887
.966
.901
.974
Table 1: Comparison of the effectiveness of different methods. We highlight that our proposed GOLD model outperforms all baselines across six data sets and both metrics. We denote best results bold, while best result among the competing methods marked with underline. Further analysis is provided in Section 6.1.
al., 2013), DistMult (Yang et al., 2015), Com- plEx (Trouillon et al., 2016), and Rotatel (Sun et al., 2019); (ii) embedding-based methods that are aware of noise, including CKRL (Xie et al., 2018) and CAGED (Zhang et al., 2022b); (iii) lan- guage model-based methods that encode both se- mantic and structural embeddings and are unaware of noise, including KG-BERT (Yao al., 2019) and LASS (Shen et al., 2022). KGist (Belth al., 2020) as rule-based method requires node type information, which is unavailable in the CSKG, making infeasible to use as a baseline. More detailed descriptions are in Appendix A.2.
Experiments and Analyses
6.1 Main Results
The performance of all models on the six datasets in the noise detection task is shown in Table 1. In general, GOLD can detect noise in CSKG more accurately, outperforming all bascline methods by large margin. Unlike baseline models based on language models, whose performance significantly increases with the size of the language model, our GOLD method consistently surpasses the baseline across different language model backbones with small performance variation. Specifically, when us- ing the RoBERTa family of language models, our GOLD method achieves average accuracy im- provement 8.64% and 8.50% compared to LASS methods on the ConceptNet and ATOMIC dataset series, respectively. Among the language models we use, the Sentence- T5-xxl model exhibits the best overall performance, with the highest accuracy improvement over 10. 14.14% and 9. 9.17% on the Con- ceptNet and ATOMIC dataset series, respectively, compared to the baseline. Additionally, the AUC score also improves by 1.02% and 0.62%.
5.4 Implementation Details
We leverage three families of PLMs from the Huggingface Library (Wolf al., 2020) to build our GOLD framework, including RoBERTa (Liu et al., 2019), DeBERTa-v3 (He al., 2023), and Sentence-T5 (Ni et al., 2022). Detailed variants of these PLMs are included in Table 1. We train GOLD with an Adam (Kingma and Ba, 2015) opti- mizer, with the learning rate set to 1e-3. The default number of training epochs is 10, with margin 7 of 5 and rule length set to 3. Additionally, we con- duct grid search for ranging from 0 to 1, find the best hyperparameter for kyuics from to 500. Further information regarding the implementation is discussed in Appendix A.3.
6.2 Ablation Study
In this section, we conduct an ablation study on the ConceplNet-N10 dataset to evaluate the contri- bution of each component in our proposed model.

---

Model
@
AUC
GOLD (Sent-T5-xxl)
0.859
0
N10
w/o LM
0,810 5. 5.7%)
0,968(} 1.3%)
0.00
w/o Egusal
0.826[L 3.8%
0,971(L 1.0%)
Reall
w/o Fsocal
0.599( 30.1%)
0.925() 5.7%)
0.84
Table 2: Ablation study results comparing the perfor- mance of GOLD with and without each component on the ConceptNet-N10 dataset.
Number Scletted Balos
Figure 3: Accuracy of noise detection vs kyules: the number of selected logical rules ConceptNet series.
The results of this study are presented in Table Overall, we observe that removing any of the com- ponents results in varying degrees of performance degradation, emphasizing the essentiality of each component in our GOLD model.
Model
Recall@
AUC
GOLD (Sentence-115xxx))
0.869
0.982
GPT-3.5 (text-davinci-003)
0.273
0.685
ChatGPT (gpt-3.5-turbr
0.263
0.734
Table 3: Performance comparison randomly sam-
Influence of Language Model We remove the PLM from the triple encoder and use random em- beddings to encode the information of nodes and relations, obtaining the embeddings 8h, 8r, 8; in Equation (1). This results in a 5.7% decrease in the model's accuracy and 1.3% decrease in AUC. indicating that the PLM indeed contributes to derstanding the semantic information of nodes. is worth noting that even after removing the language model, the accuracy and AUC still outperform all competing methods.
pled ConceptNet-N10 dataset.
Influence of Local Neighbor Learning More- over, remove the local neighbor information learning component, resulting significant de- crease of 30.1% accuracy and 5.7% in AUC, demonstrating the crucial role neighboring triple information in noise detection. More comprehen- sive ablation studies are in Appendix C.
6.3 Comparison with ChatGPT
Influence of Global Rule Mining We remove the global rule encoder, which results in 3.8% decrease in accuracy and 1.0% decrease in AUC, implying the important role of the rule encoder in guiding noise detection. Furthermore, as we train the rule encoder using the top krules rules with the highest confidence score for each relation from the rules mined by AMIE 3, we test the impact of different values of krutes on the accuracy us- ing three datasets from the ConceptNet series. We vary krules among {100, 200, 300, 400, 400,500) 500). The results are shown in Figure We observe that when the noise level is relatively low, i... in the N5 dataset, kyules = 200 achieves the best perfor- mance, and adding more rules degrades the model's performance. However, increasing the number of rules improves the model's performance to some extent when the noise level is high, such in the N10 and N20 datasets. We analyze that this is be- cause as the noise level increases, learning local information becomes more prone to being misled. Hence, more rules are needed to provide global guidance.
Recent breakthroughs in Large Language Mod- els (LLMs), such GPT-3.5 (Brown al., 2020; Ouyang et al., 2022) and ChatGPT (OpenAI, 2022). have demonstrated remarkable performance across a diverse range of NLP tasks (Chan et al., 2023; Qin et al., 2023). In light of this, we benchmark these LLMs on our defined noise detection task to establish another competitive baseline for compari- son. To accomplish this, we randomly select 1,000 triples from our poisoned ConceptNet-N10 CSKG and ask the LLMs rank them by iteratively com- paring two triples and merge- them (more detailed information in Appendix B). This eval- uation setting ensures that the LLMs follow an objective that is mostly identical to GOLD. The re- sults, shown in Table 3, indicate that both LLMs perform significantly poorly our task, leaving substantial gap compared to GOLD. One pos- sible explanation is that these LLMs operate in a zero-shot setting and lack prior knowledge of noisy knowledge contained CSKGs. This highlights the significance of GOLD, which exhibits a keen sensitivity to noise CSKGs through fine-tuning.

---

6.4 Downstream Benefits of Denoising CSKG
Denoising
aNLI
CSQA
PIQA
SIQA
WG
Avg.
N/A
74.0
65.4
73.8
59.5
73.9
69.3
We finally validate the effectiveness of our pro- posed noise detection framework by investigat- ing whether climinating noise from ATOMIC would yield extrinsic benefits for downstream tasks, specifically, zero-shot commonsense Question- Answering (QA) (Ma et al., 2021). This task in- volves performing QA commonsense bench- marks, such as Abductive NLI (aNLI; Bhagavat- ula al., 2020). CommonsenseQQ (CSQA: Tal- mor al., 2019), PhysicalIQA (PIQA: Bisk al., 2020), SocialIQA (SIQA; Sap et al., 2019b), and WinoGrande (WG; Sakaguchi et al., 2021), with- out accessing their respective training data. Ma et al. (2021) proposed technique that fine-tunes a PLM on synthetic QA pairs constructed from CSKGs, which has been further improved by Kim al. (2022) using modularized transfer learn- ing and Wang et al. (2023a) with conceptualiza- tions (Wang al., 2023b). Specifically, the head node and relation of an edge are transformed into a question using natural language templates, and the tail node serves as the ground-truth answer. Distractors are tails of other edges sampled from the same CSKG whose head node does not share common keywords with the question. A PLM is then fine-tuned on such synthetic QA entries us- ing marginal ranking loss serve as general QA model. To this extent, we keep the QA synthesis protocol and model training process fixed and abla- tively study the role of leveraging different CSKGs, in our case, raw 100 and noise-cleaned ATOMIC 10X We use accuracy the evaluation metric and trained three QA models separately (1) the original ATOMIC 10x (2) ATOMIC 10x de- noised with LASS, and (3) ATOMIC denoised with GOLD, where the former two served as the baselines. The results are reported in Table 4. We observe that cleaning ATOMIC!0R 10x with GOLD out- performs both baselines on average, indicating that denoising CSKG is potentially useful for automati- cally generated CSKGs and that GOLD is superior to other noise detection frameworks on real-world CSKGs.
LASS
71.8
63.8
77.7
57.4
67.3
68.0
GOLD
72.2
69.6
79.0
58.8
71.5
70.3
Table 4: Zero-shot evaluation results (%) on five bench- marks for QA models trained on the original/denoised ATOMIC N/A stands for not using any denoising technique, and Avg. refers average.
Conclusions
In this paper, propose GOLD, noise detection framework leveraging the power of language mod- els, global rules, and local structural information. This method is motivated by the fact that nodes in CSKGs are in free-text format, and correct patterns are unlikely to be drowned out by noise. Experi- mental results indicate that our method achieves state-of-the-art performances in CSKG noise de- tection tasks. This method shows promising di- rections for automatically obtaining large-scale CSKG with minimal noise, as well as effectively representing knowledge for downstream tasks.
Limitations
In our experiments, we follow the approach of pre- vious noise detection literature (Xie et al., 2018; Jia et al., 2019) and inject synthesized noise back into the original CSKGs. Although this noise injection technique has been deemed reliable in previous works, further investigation necessary to verify its rigor in the field of commonsense reasoning. This is because such noise can typically be classi- fied as negative commonsense knowledge, which, as suggested by Chen et al. (2023), should be ver- ified by whether it can be grounded as negative knowledge. Alternatively, we could inject noise from the perspective of graph attacks (Zhang al., 2019a) to increase the difficulty of noise detection and improve the model's robustness.
Ethics Statement
This paper introduces GOLD, a novel denoising framework for CSKG noise detection that is both global and local-aware. The experiments presented in this paper utilize open-source datasets, includ- ing ConceptNet, ATOMIC, ATOMIC and five commonsense question-answering benchmarks. The crowdsourced datasets, such as ConceptNet, ATOMIC, and the five commonsense question- answering benchmarks, have been manually cu-
6.5 Case Study
We present specific case studies on the mined logi- cal rules and detected noises the real large-scale CSKG in Appendix D. Those cases directly show the effectiveness of our proposed method.

---

rated and fürther processed ensure that they are anonymized and desensitized. The experiments align with their intended usage, which is for re- search purposes. Additionally, while ATOMIC is generated using language models, its prompt engineering ensures that no harmful content is gen- erated, which has been verified by manual inspec- tion (West et al., 2022). Therefore, to the best of the authors' knowledge, we believe that GOLD in- troduces no additional risk.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432- 7439. AAAI Press.
Antoine Bordes, Nicolas Usunier, Alberto Gareía- Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Pro- ceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 2787-2795.
Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable feedback. The au- thors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20), and the GRF (16211520 and 16205322) from RGC of Hong Kong. We also thank the UGC Research Match- ing Grants (RMGS20EG01-D, RMGS20CR11, I,, RMGS20CR12, RMGS20EG19, RMGS20EG21, RMGS23CR05, RMGS23EG08).
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chai- tanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019. COMET: commonsense transformers for auto- matic knowledge graph construction. In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4762 4779. Association for Computational Linguis- tics.
References
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemerr, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual.
Bo An, Bo Chen, Xianpei Han, and Le Sun. 2018. Ac- curate text-enhanced knowledge graph representa- tion learning. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2018, New Or- leans, Louisiana, USA, June 1-6, 2018, Volume I (Long Papers), pages 745-755. Association for Com- putational Linguistics.
Jiaxin Bai, Xin Liu, Weiqi Wang, Chen Luo, and Yangqiu Song. 2023. Complex query answering on eventuality knowledge graph with implicit logical constraints. CoRR, abs/2305. 19068.
Chunkit Chan, Jiayang Cheng, Weiqi Wang, Yuxin Jiang, Tianqing Fang, Xin Liu, and Yangqiu Song. 2023. Chatgpt evaluation sentence level relations: A focus on temporal, causal, and discourse relations. CoRR, abs/2304. 148277.
Molood Barati, Quan Bai, and Qing Liu. 2017. Mining semantic association rules from RDF data. Knowl. Based Syst., 133:183-196.
Caleb Belth, Xinyi Zheng. Jilles Vreeken, and Danai Koutra. 2020. What is normal, what is strange, and what is missing in a knowledge graph: Unified char- acterization via inductive summarization. In WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 1115-1126. ACM/ / IW3C2.
Jiangjie Chen, Wei Shi. Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. 2023. Say what you mean! large language models speak too positively about negative commonsense knowledge.
Kewei Cheng, Nesreen K. Ahmed, and Yizhou Sun. 2023. Neural compositional rule learning for knowl- edge graph reasoning. CoRR, abs/2303.03581.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Represen- tations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. Oee
Kewei Cheng, Jiahao Liu, Wei Wang, and Yizhou Sun. 2022. Rlogic: knowledge graphs. In KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and
Recursive logical rule learning from

---

Data Mining, Washington, DC, USA, August 14 18. 2022, pages 179-189.14 ACM
Xu Han, Shulin Cao, Xin Lv. Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. 2018. Openke: An open toolkit for knowledge embedding. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: Sys- tem Demonstrations Brussels, Belgium, October 31 November 4, 2018, pages 139-144, Association for Computational Linguistics.
Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in artificial intelligence. Commun. ACM, 5899)922-
Tianging Fang, Quyet V. Do. Schyun Choi, Weiqi Wang. and Yangqiu Song. 2023. CKBP v2: An expert- annotated evaluation set for commonsense knowl- edge base population. CoRR, abs/2304.10392.
Mutian He, Tianqing Fang. Weiqi Wang, and Yangqiu Song. 2022. Acquiring and modelling abstract com- monsense knowledge via conceptualization. CoRR, abs/2206.01532.
Tianqing Fang. Weiqi Wang. Sehyun Choi, Shibo Hao, Hongming Zhang. Yangqiu Song, and Bin He. 202 la Benchmarking commonsense knowledge base pop- ulation with an effective evaluation dataset. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2021. Virtual Event/ / Punta Cana, Dominican Republic, 7- 11 November, 2021, pages 8949-8964. Association for Computational Linguistics.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTav3: Improving deBERTa using ELECTRA- style gre-training with gradient diseentnddddddeeeddd embed- ding sharing. In The Eleventh International Confer- ence on Learning Representations.
Shengbin Jia, Yang Xiang, Xiaojun Chen, Kun Wang. and Shijia E. 2019. Triple trustworthiness measure- ment for knowledge graph. In The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pages 2865-2871, ACM.
Tianqing Fang, Hongming Zhang, Weiqi Wang, Yangqiu Song, and Bin He. 202 DISCÓS: hridg- ing the gap between discourse knowledge and com- monsense knowledge. In WWW 21: The Web Con- ference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, pages 2648-2655 ACM/ / IW3C2.
Yu Jin Kim, Beong-woo Kwak. Youngwook Kim. Reinald Kim Amplayo, Seung-won Hwang, and Jiny- oung Yeo. 2022. Modularized transfer learning with multiple knowledge graphs for zero-shot common- sense reasoning. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguisties: Human Language Technologies. NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 2244-2257. Association for Computational Linguistics.
Luis Galárraga, Christina Teffioudi, Katja Hose, and Fabian M. Suchanek. 2015. Fast rule mining in on- tological knowledge bases with AMIE+. VLDB J., 24(6):707-730.
Luis Antonio Galárraga, Christina Teffiodii Katja Hose. and Fabian M. Suchanek. 2013. AMIE: association rule mining under incomplete evidence ontologi- cal knowledge bases. In 22nd International World Wide Web Conference, WWW '13, Rio de Janeiro Brazil, May 13-17, 2013, pages 413-422. Interna- tional World Wide Web Conferences Steering Com- mittee/ / ACM.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Congcong Ge. Yunjun Gao, Honghui Weng, Chong Zhang, Xiaoye Miao, and Baihua Zheng. 2020. Kg- clean: An embedding powered knowledge graph cleaning framework. CORR, hbs/2004.144753
Jonathan Lajus, Luis Galárraga, and Fabian M. Suchanck. 2020. Fast and exact rule mining with AMIE 3. In The Semantic Web 17th International Conference, ESWC 2020, Heraklion, Crete, Greece, May 31-June 4, 2020, Proceedings, volume 12123 of Lecture Notes in Computer Science, pages 36-52. Springer.
Shu Guo, Quan Wang. Lihong Wang, Bin Wang, and Li Guo. 2016. Jointly embedding knowledge graphs and logical rules. In Proceedings of the 2016 Confer- ence on Empirical Methods in Natural Language Pro- cessing, EMNLP 2016, Austin, Texas, USA, Novem- ber 1-4, 2016, pages 192-202. The Association for Computational Linguistics.
Xiang Li, Aynaz. Taheri, Lifu Tu, and Kevin Gimpel. 2016. Commonsense knowledge base completion. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Shu Guo, Quan Wang, Lihong Wang, Bin Wang, and Li Guo. 2018. Knowledge graph embedding with iterative guidance from soft rules. In Proceedings of the Thirry-Seoond AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applica- tions of Artificial Intelligence (1AAI-18), and the 8th AAAI Symposium on Educational Advances in Artifi- cial Intelligence (EAAI-18), 18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4816 AAAI Press.
Yankai Lin, Zhiyuan Liu, Huan-Bo Luan, Maosong Sun, Siwei Rao, and Song Liu. 2015. Modeling rela- tion paths for representation learning of knowledge bases. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 705-714. The Association for Computa- tional Linguistics.

---

H. Liu and Singh. 2004, Conceptnet a practical commonsense reasoning tool-kit. BT Technology Journal, 22(4):211-226
Jianmo Ni. Gustavo Hernández Abrego. Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. 2022. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the As- sociation for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1874.1874 Association for Computational Linguisties.
Rui Liu, Zheng Lin, Yulong Tan, and Weiping Wang.
2021. Enhancing zero-shot and few-shot stance de- tection with commonsense knowledge graph. In Findings of the Association for Computattonnl Lin- guisties: ACL/JJCNLP 2021, Online Event, August 1-6, 2021, volume ACI/UJNLPP 2021 of Findings of ACL, pages 3152-3157. Association for Computa- tional Linguistics.
OpenAL. 2022. Chatgpt: Optimizing language models for dialogue. OpenAl.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NeurIPS.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 1692.
Kaixin Ma, Filip Ilievski. Jonathan Francis, Yonatan Bisk, Eric Nyberg. and Alessandro Oltramari. 2021. Knowledge-driven data construction for zero-shot evaluation in commonsense question answering. In Thirty-Fifth AAAI Conference on Artificial Intelli- gence, AAAI 2021, Thirty- Conference on In- novative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2021, Vir- tual Event, February 2-9, 2021, pages 13507-13515. MAAIPres.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang. Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purpose natural language process- ing task solver? CoRR, abs/2302.06476.
Meng Qu, Junkun Chen, Louis- A. C. Xhonneux, Yoshua Bengio, and Jian Tang. 2021. Rnnlogic: Learning logic rules for reasoning on knowledge graphs. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OvenReview.net.
Chaitanya Malaviya, Chandra Bhagavatula, Antoine Bosselut, and Yejin Choi. 2020. Commonsense knowledge base completion with structural and se- mantic context. In The Thirty-Fourth AAAI Con- ference on Artificial Intelligence, AAAI 2020, The Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial In- telligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 2925-2933, AAAI Press.
Julien Romero and Simon Razniewski. 2023, Mapping and cleaning open commonsense knowledge bases with generative translation.
Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. 2019. DRUM: end-to- end differentiable rule mining on knowledge graphs. In Advances in Neural Information Processing Sys- tems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 15321- 15331.
Elan Markowitz, Keshav Balasubramanian, Mchrnoosh Mirtaheri, Murali Annavaram, Aram Galstyan, and Greg Ver Steeg. 2022. Statik: Structure and text for inductive knowledge graph completion. In Find- ings the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 604-615. Association for Computational Linguistics.
Tara Safavi, Jing Zhu, and Danai Koutra. 2021. Negater: Unsupervised discovery of negatives in common- sense knowledge bases. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing. EMNLP 2021, Virtull Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 5633-5646. Association for Computa- tional Linguistics.
Nasrin Mostafazadeh, Aditya Kalyanpur, Lori Moon, David W. Buchanan, Lauren Berkowitz, Or Biran, and Jennifer Chu-Carroll 2020. GLUCOSE: gen- eralized and contextualized story explanations. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 4569 4586. Association for Computational Linguistics.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2021. Winogrande: an adver- sarial winograd schema challenge at scale. Commun. ACM, 64(9):99- 1066
Maarten Sap, Ronan Le Bras, Emily Allaway, Chan dra Bhagavatula, Nicholas Lourie, Hannah Rashkin. Brendan Roof, Noah A. Smith, and Yejin Choi 2019a. ATOMIC: atlas of machine commonsense for if-then reasoning. The Thirt- Third AAAI Con- ference on Artificial Intelligence, AAAI 2019, The
Stephen H. Muggleton and Lue De Raedt. 1994, Induc- tive logic programming: Theory and methods. J. Log. Program., 19/20:629-679.

---

Thirty-First Innovative Applications of Artificial In- telligence Conference, LAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- - February 1, 2019, pages 3027-3035. AAAI Press.
Niket Tandon, Gerard de Melo, and Gerhard Weikum. 2014. Acquiring comparative commonsense knowl- edge from the web. In Proceedings of the Twenty- Eighth AAAI Conference on Artificial Intelligence, July 27 -31, 2014, Québec City, Québec, Canada, pages 166-172. AAAI Press.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social iqa: Common- sense reasoning about social interactions. In Proceed- ings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing, EMNL.P-JCNIPP 2019, Hong Kong. China, November 3-7, 2019. pages 4462 4472. Association for Computational Linguistics.
Niket Tandon, Gerard de Melo, and Gerhard Weikum. 2017. Webchild 2.0 Fine-grained commonsense knowledge distillation. In Proceedings of the 55th Annual Meeting the Association for Computational Linguistiex, ACL 2017. Vancouver, Canada, July August 4, System Demonstrations, pages 15-1200. Association for Computational Linguisties.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Com- plex embeddings for simple link prediction. In Pro- ceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Work- shop and Conference Proceedings, pages 2071-2080. JMLR.org.
Mike Schuster and Kuldip K. Paliwal. 1997. Bidirec- tional recurrent neural networks. IEEE Trans. Signal Process, 45(11):2673-2811
Jianhao Shen, Chenguang Wang, Linyuan Gong, and Dawn Song. 2022. Joint language semantic and structure emhedding for knowledge graph comple- tion. In Proceedings of the 29th International Confer- ence on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 1965-1978. International Committee on Com- putational Linguistics.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3. 2018, Conference Track Proceedings. OpenReview.net.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 4444 4451, AAAI Press.
Bin Wang, Guangtao Wang, Jing Huang, Jiaxuan You. Jure Leskovec, and C.-C. Jay Kuo. 2021a. Inductive learning on commonsense knowledge graph comple- tion. In International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021, pages 1-8, IEEF.
Ying Su. Zihao Wang. Tianqing Fang, Hongming Zhang, Yangqiu Song. and Tong Zhang. 2022. MICO: A multi-alternative contrastive learning framework for commonsense knowledge representation. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, De- cember 7-11, 2022, pages 1339-1351. Association for Computational I Linguistics.
Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Ying Wang, and Yi Chang. bb2 firucture-augmentt text representation learning for efficient knowledge graph completion. In WWW '21: The Web Confer- ence 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, pages 1737-1748.
Weiqi Wang. Tianqing Fang, Wenxuan Ding, Baixuan Xu, Xin Liu, Yangqiu Song, and Antoine Bosse- lut. 2023a. CAR: wtwppptaaaiiaiiinuuneeeeee soner for zero-shot commonsense question answer- ing, CoRR, abs//335511488
rea-
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. In 7th In- ternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Weiqi Wang, Tianqing Fang, Baixuan Xu, Chun Yi Louis Bo, Yangqiu Song, and Lei Chen. 2023b. CAT: A contextualized conceptualization and instan- tiation framework for commonsense reasoning. In Proceedings of the 61 Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Pupers), ACI Torunio, Canada, July 9-14, 2023, pages 13111-13140. Association for Computa- tional Linguistics.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowl- odge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACI-HHL 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2149-4158. Association for Computational Linguistics.
Zhaowei Wang, Quyet V. Do, Hongming Zhang, Jiayao Zhang, Weiqi Wang. Tianqing Fang, Yangqiu Song, Ginny Y. Wong, and Simon See. 2023c. COLA: con- textualized commonsense causal reasoning from the

---

causal inference perspective. In Proceedings of the 61st Annual Meeting of the Association for Compu- national Linguistics (Volume 1: Long Pupers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 5253- 5271. Association for Computational Linguistics.
Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 23 19-2888.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. KG-BERT: BERT for knowledge graph completion. CoRR, abs/1909.03193.
Zhigang Wang and Juan-Zi Li. 2016. Text-enhanced representation learning for knowledge graph. In Pro- International Joint Con-
ceedings of the Twenty-Fii ference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 1293-1299. LICALAААI Press.
Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy Liang. and Jure Leskovee. 2022. Deep bidirectional language-knowledge graph pretraining, In NeurlPS.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le. and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NeurlPS.
Changlong Yu, Weiqi Wang, Xin Liu, Jiaxin Bai, Yangqiu Song. Zheng Li, Yifan Gao, Tianyu Cao. and Bing Yin. 2022. Folkscope: Intention knowl- edge graph construction for discovering e-commerce commonsense. CoRR, abs/2211 1.00226......6..
Peter West, Chandra Bhagavatula. Jack Hessel, Jena D. Hwang. Liwei Jiang. Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language mod- els to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, NAACL 2022, Seattle WA, United States, July 10-15, 2022, pages 4602- 4625. Association for Computational Linguistics.
Hengtong Zhang, Tianhang Zheng, Jing Gao, Chenglin Miao, Lu Su, Yaliang Li, and Kui Ren. 2019a. Data poisoning attack against knowledge graph embed- ding. In Proceedings of the Twenty-Eige Eighth Interna- tional Joint Conference on Artificial Intelligence, IJ- CAI 2019, Macao, China, August 10-16, 2019, pages 4853 4859. ijeai.org.
Hongming Zhang, Xin Liu, Haojie Pan, Haowen Ke, Jiefu Ou, Tianqing Fang, and Yangqiu Song. 2022a. ASER: towards large-scale commonsense knowl- edge acquisition via higher-order selectional pref- erence over eventualities. Artificial Intelligence, 309:103740
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: saat--of-the-ate natural language processing. Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing: System Demon- strations, EMNLP 2020 Demos, Online, November 16-20, 2020, pages 38-45. Association for Computa- tional Linguisties.
Houyu Zhang, Zhenghao Liu, Chenyan Xiong, and Zhiyuan Liu. 2020. Grounded conversation genera- tion as guided traverses in commonsense knowledge graphs. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 2031-2043. Association for Computational Linguistics.
Qinggang Zhang, Junnan Dong, Keyu Duan, Xiao Huang, Yezi Liu, and Linchuan Xu. 2022b. Con- trastive knowledge graph error detection. In Proceed- ings of the 31 ACM International Conference on Information & Knowledge Management, Atlania, GA, USA, October 17-21, 2022, pages 2590-2599, 2599. ACM.
Ruobing Xie. Zhiyuan Liu, Fen Lin, and Leyu Lin. 2018. Does william shakespeare REALLY write hamlet? knowledge representation learning with confidence. In Proceedings of the Secooy AAAI Confer- ence on Artificial Intelligence, (AAAI-18), the 30th in- novative Applications of Artificial Intelligence (IAAI- 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (FAAI- 18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4954 4961. AAAI Press.
Wen Zhang, Bibek Paudel, Liang Wang. Jiaoyan Chen, Hai Zhu, Wei Zhang, Abraham Bernstein, and Hua- jun Chen. 2019b. Iteratively learning embeddings and rules for knowledge graph reasoning. In The World Wide Web Conference, WWW 2019, San Fran- cisco, CA, USA, May 13-17, 2019, pages 2366-2377. ACM.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Fan Yang, Zhilin Yang, and William W. Cohen. 2017. Differentiable learning of logical rules for knowledge base reasoning. In Advances in Neural Information

---

Appendices
statistical information for the datasets is presented in Table 5.
A Experimental Setup Details
A.2 Competing Methods
A.1 Datasets
We compare GOLD with three categories of algo- rithms, beginning with four structure embedding- based methods that are unaware of noise. Here, h, r represent the embeddings of the head entity, relation, and tail entity, respectively.
ConceptNet ConceptNet, or CN-100K, was first proposed by (Li et al., 2016). It contains Open Mind Common Sense (OMCS) in the ConceptNet 5 dataset. CN-82K dataset (Wang et al., 202 Ia) is a uniformly sampled version of the CN-100K dataset.
TransE (Bordes et al., 2013) The score func- tion is ||h + t||, where h, r, E Rd.
ATOMIC ATOMIC contains over 300K everyday commonsense knowledge nodes, organized as if- then relations. It proposes nine types of if-then rela- tions distinguish various aspects of events, such causality, intents, and mental states. Malaviya et al. constructed a dataset from ATOMIC for the task of CSKG completion.
DistMult (Yang et al., 2015) The score func- tion is (r, h, h)t), where (-) denotes the gener- alized dot product, and h, r, t E Rd.
ComplEx (Trouillon et al., 2016) The score function is 9 ((r, ; h, t)), where h, r, t E Cd.
In our experiments, we follow Wang et al. (2021a) to use CN-82K and ATOMIC. Unlike CSKG completion settings, merge the train, valid, and test split get training and testing sets because noise detection is a ranking task requir- ing training and testing on the entire knowledge graph. To introduce noisy triples, we follow Xie al. (2018) and Jia et al. (2019) to add noisy triples to these two datasets separately manually. Specif- ically, the noise we generate is divided into four parts, with probability of 1/4 for randomly gener- ating new triple (h, f, i) where h,t E V, f E R. and probabilities of 1 1/4 each for modifying the head node, relation, or tail node of an existing triple. When modifying an existing triple, we randomly sample a ground truth triple (h, r, t) E from the CSKG and then replace one of its components
RotatE (Sun et al., 2019) The score function is /hor-ttl3 where denotes the Hadamard product, and h, r, E Cd.
Next, we consider two embedding based meth- ods that capture noise using local information:
CKRL (Xie et al., 2018) They introduce the triple confidence and path confidence to conventional translation-basee methods for knowledge representation learning.
CAGED (Zhang et al., 2022b) They pro- pose a contrastive learning framework to cap- ture noise by aggregating triple information around the head and tail entities while also learning the traditional translation embedding.
with randomly chosen node à. f c V. or relation
We also evaluate our methods against fine-tuned language models:
f E RR to create new triple (h, rr, l), (h, f, () or (h, rr,)) f). The process of generating noisy triples requires ensuring that they do not exist the orig- inal CSKG. Taking (hotel room, UsedFor, tempo- rary residence) from ConceptNet and (John works long hours, xIntent, to make more money) from ATOMIC as examples, Table 6 presents several ex- amples of noise generated by replacing the head node, relation, and tail nodes, well as examples of newly generated triples. It can be observed that these noises are still informative and theoretically challenging to detect, aligning with our previous definition of noises in CSKG in Section 3. Hence, we believe that the noise generated through the above method is effective for model training. The
KG-BERT (Yao et al., 2019) They first pro- pose concatenating the triples into textual de- scriptions and transforming the representation learning into a triplet classification problem. We evaluate the performance of noise detec- tion by using scores designed for classifica- tion.
LASS (Shen et al., 2022) They propose a joint language semantic and structure embedding for knowledge graph completion. We also use the scores designed for triplet classification to evaluate the performance. Experimental results from their paper demonstrate that their

---

Dataset
Nodes
Relations
Original Triples
Noisy Triples
Avg. Degree
Avg. Words
Con-eptNet-N5
5,120
ConceptNet-N10
78,339
34
102,400
10,240
1.31
2.85
ConceptNet-N20
20,480
ATOMIC-N5
39,297
ATOMIC-NI0
304,439
9
762,230
78,595
2,50
4.47
ATOMIC-N20
157,190
Table 5: Statistical information for six datasets. Avg. Degree represents the average degree of each node and Avg. Words represents the average numher of words in the text description of each node.
Type
Head
Relation
Tail
playground equipment
UsedFor
temporary residence to make more money
Replacing & with h
John has trouble falling asleep
xIntent
hotel room
NotCapableOf
temporury residence to make more money
Replacing with
John works long hours
offfect
Replacing with i
hotel room
UsedForr
prepare food to eat lose money or resources
John works long hours
xIntent
plastic fork
CapableOf
buy food
New Triple
John drinks coffee
oEffect
to go to the movie theatre
Table Examples of four manually generated types noise. The ground truth triples modified in the examples are (hotel room, UsedFor, temporary residence) from ConceptNet and (John works long hours, xIntent, to make more money) from ATOMIC. The modified parts are indicated by underlines.
model outperforms other PLM-based methods in triplet classification tasks. Hence, select as our baseline. In particular, their model is tested on four backbones, namely BERT-baase, BERT-large, RoBERTa-base and RoBERTa- large. We also conduct experiments on these four backbones.
(2023c) and call the ASKCHATGPT function em- ploy ChatGPT choose which triple to be noisy from two triples. Inspired by chain-of- thought (CoT) prompting (Wei al., 2022), we guide ChatGPT in the prompt to first provide the specific reasoning process and then compel it to provide the answer. The prompt used for compar- ing which of the two triples is more likely to be noise is listed in Table 7. We use OpenAl's API to prompt ChatGPT and retrieve its response.
likely
A.3 Implemention Details
For the embedding- -bsed baseline models, we use the implementation from OpenKE(Han et al., 2018). For the rest, we use the released code corre- sponding to each paper to perform experiments. In order to align the performance of different models, the dimension of all embeddings apart from language models 100, the number of negative samples 1, and the batch size 256. Our model also follows these settings. For the remaining hy- perparameters of baseline models, we follow the settings proposed in the original paper and perform a grid search when modifications are necessary.
Prompt
Given two triples from a knowledge graph: (1,, (2,, Which one more likely to be wrong? Show me the reason first, and then print the wrong triple. You are forced to make a decision.
Table 7: A natural language prompt used to guide Chat- GPT to compare two triples and determine which one is more likely to be noise. Entries in italics will be placed by actual triples. The last sentence mandates ChatGPT choose which of the two triples more likely to be noise.
B Details of the Zero-shot Noise Detection
ChatGPT cannot directly sort large number of triples, so we implement a merge sort in Algo- rithm sort the triples in descending order of their noise level. When comparing the order of two triples, we draw inspirations from Wang et al.
C Full Results of Ablation Study
In this section, we provide comprehensive sup- plementary ablation study. The results of all exper-
hhtps://chat.ppema..co/

---

ConceptNet
Model
N5
N10
N20
AUC
Acc
AUC
Acc
AUC
GOLD (Sent-T5-xxl)
.842
.985
.859
.981
.878
.979
w/o PLM
.779(L 7.5%)
.970(1 1.5%)
8800 5.7%)
.988CU 1.3%)
.834(L 5.0%)
.63(C) 1.6%)
w/o Eguubal
.791(] 6.1%)
.74() 1.1%)
826([] 3.8%)
.71(L 1.0%)
.622(] 1.8%)
974(L 0.5%)
w/o Emasl
.478(L 43.2%)
.95( 7.1%)
.599(] 30.3%)
.25(() 5.7%)
.652() 25.7%)
.07(L 7.4%)
w/ Pinasiation
841() 0.1%)
.866(1 0.1%)
886()) 0.4%)
.83(( 0.2%)
867( 1.3%)
971() 0.8%)
ATOMIC
Model
N5
N10
N20
Acc
AUC
Acc
AUC
Acc
AUC
GOLD (Sent-T5 T
.872
.969
.887
.966
.901
.974
w/o PLM
.779() 10.7%)
.27( 4.3%)
881({ 9.7%)
.92(CU 3.9%)
.822(L 8.8%)
.929(1 4.6%)
w/o Egioral
859C 1.5%)
.900(] 0.9%)
.874(] 1.5%)
.91(C) 0.5%)
.884Q 1.9%)
.955L] 2.0%)
w/o Emant
.656(] 24.8%)
.31( 3.9%)
.699(] 21.2%)
.300 3.7%)
.747() 17.1%)
.966(] 4.9%)
Eirastation
868(L 0.5%)
.622( 0.7%)
88664 0.1%)
.95(LL 0.1%)
.911C 0.0%)
.700 0.4%)
Table 8: The comprehensive ablation study results comparing the impact of cach component on the results all six datasets. Additionally, we verify the effect of adding the translation- baasad energy function on the results.
Algorithm 1 Merge Sort guided by ChatGPT
ConceptNet. Therefore, PLM plays more crucial role in capturing semantic information.
Input: A triple list L
Output: A tiple list sorted from high to low according to the noise level
Influence of Global Rule Mining After elimi- nating the global rule encoder, the accuracy of the ConceptNet series and ATOMIC series datasets de- creases by 3.9% and 1.6%, respectively. Our analy- sis suggests that the lower number of relations in the ATOMIC datasets, only compared to 34 in the ConceptNet datasets, results in significantly lower number of learnable rules compared the ConceptNet. As result, the global rule encoder provides limited assistance the ATOMIC datasets, and its contribution is not as significant as the ConceptNet datasets.
Function: MERGESORT(L))
+ 111/2
2: Lieft + 1, ..
3: Lright + MERGESORI(I 1, L|)
4: i+1
5: j 4-1
6: for ls f to [L] do
7:
if > h then
8:
L[k] & Liight[i]]
9:
j +
10:
else if j > h then
11:
L[k] + Liess[i]
12:
i + i+1
13:
else if ASK (LKCHATTPT(LL, L2) = L. then
14:
[[K] + [i)
15:
i +-i+1
16:
else
Influence of Local Neighbor Learning The lo- cal neighbor learning component exhibits the high- est contribution across all datasets, evidenced by the average accuracy drops of 33.1 % and 21.0% in accuracy, as well as 6.7% and 4.2% in AUC af- ter removal on ConceptNet series and ATOMIC series datasets, respectively. We believe that the reason why this component has a smaller impact on the ATOMIC datasets is still due to the limited number of relations, leading to less diverse set of information learned from the neighboring triple information.
17:
L[k] +- Liighalil
18:
+ i
19:
end if
20:
end for
21:
return L
iments conducted the six datasets are listed in
Table 8.
Influence of Language Model By removing the PLM from the triple encoder, we observe an aver- age decrease of 6.1 % in accuracy the Concept- Net series datasets and average decrease of 9.7% on the ATOMIC series datasets. This indicates that PLM has a greater impact on the accuracy of the ATMOIC datasets, as the average number of words per node in ATOMIC much higher than that in
Influence of Translation Assumption We tempt to investigate whether the model would ben- cfit from the incorporation tion, such as the + r z t relation in TransE (Bor-
translation assump-

---

Rules
IsA(æ, Is,( =1) DefinedAs(zi, 22) IsA(=2, y) (2pable0f(2; u) + (2; Parto A Capabledf(22.
(ot y) DefinedAs(r; a) CreatedBy(#1 NotDesires(zz HasPrerequisite(r y) HasPrerequisiteez; 51) MadeOf (21, 22) HasPrerequisite((zz.))
Table 9: Examples of the most frequent rules mined from the ConceptNet-N10 dataset.
Head
Relation
X good friend with Y
isAfter
X little weird glad to X to leave X's
X refuses to take pen
oEffect
X steals Y's breakfast
oNeed
X catches stranger
isAfter
X checks the weather forecast
X. flies Washington
oEffect
become politician
X falls down again
isAfter
X's mother gives X hug
X seems stay well
to see X get sick Y has done something that upsets X
X assumes that Y
person
isAfter
X studies hard for his exum
oNeed
to cheat on exum
X afraid getting trouble
oNeed
to tell X that or could get in trouble
Table 10: Example of noise detected in the ATOMIC 10x CSKG.
des al., 2013), where h r, represents the em- bedding of the head entity, relation, and tail entity respectively. Inspired by this, we also integrate an energy function based on the translation assump- tion into our approach. We design the energy func- tion for the translation part as follows:
observe that these rules are highly interpretable and not affected by mixed-in noise. Therefore, they can treated as ground truth validate the entire knowledge graph (Bai et al., 2023).
Detected Noise We conduct our proposed GOLD method the ATOMIC dataset and examine the triples with noise levels in the top 1 1%. We list specific examples that violate reasonability (see Section 3) Table 10. The results show that our method can effectively extract noise triples from a large-scale CSKG.
Firanslation (h, r, |le,
e:lla-
(16)
By adding Equation (16) Equation (12), ob- tain new overall energy function as follows:
E(h, r, t) = (h r.
+ XE (h, L)
(17)
+ (9) Exancitioa (h, r)),
where and (1) are both hyperparameters. We per- form a grid search for them between 0.001 to and report the best results in Table 8. The experimental results indicate that the energy function based on the translation assumption in the form of Equation (16) cannot provide significant assistance our model. The overall impact on precision is negative, with an average decrease 0.4%. This suggests that our GOLD method does not need to rely on such translation assumption constraints when per- forming noise detection task. It can implicitly learn the relationship between nodes using the energy functions of the global and local parts.
D
Case Studies
Mined Logical Rules We list the most frequent rules mined from the No1 dataset us- ing AMIE and present them in Table 9. We can


Reviews:
[{"review_id": "vuT7PQIFnH", "reviewer": "Reviewer_68ji", "summary": "", "questions": "I am only concerned about the novelty of the implementation of global and local information modeling. Have they been separately proposed in existing studies, and you are simply combining them, or is there a new approach to combining their strategies? Your explanation would help me better understand your innovative contributions and potentially raise my rating to a 4. ", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the Reviewer for providing us with valuable feedback. The following paragraphs address your reason-to-reject and question. \n\n# Shortcomings of Existing Methods\n\nThank you for bringing up the issue with the writing in the introduction. We appreciate your feedback and will make sure to improve it in the camera-ready version, following your advice. Before we do that, we would like to further clarify the shortcomings of existing methods and highlight how our method stands out.\n\nFirstly, it is important to note that nodes in CSKGs often exist in non-canonicalized and free-form text. Despite having different descriptions, these nodes can still possess related semantics. However, existing learning-based methods such as TransE and RotatE are **unable to capture the semantic information of these nodes** (lines 49-71). \nSimilarly, path-learning and neighbor-learning based methods also **fall short of capturing this semantic information** (lines 72-80). For instance, `brush your teeth` (line 72) is an isolated node in ConceptNet that cannot be distinguished based on any structural information.\nFurthermore, methods that rely on logical rules **suffer from sparsity issues and have limited generalizability** when applied to diverse rules in CSKGs (lines 81-100).\nIn the case of CSKG completion tasks, existing methods **either require extensive fine-tuning or lack sensitivity to negative commonsense knowledge**, which is crucial for noise detection (lines 111-126). \n\nOvercoming all of these challenges, we propose a novel method that addresses each of these issues with different approaches. These proposed treatments form the novelty of our method, as explained in more detail below.\n\n# Novelty of Our Method\n\nThanks for raising the issue of novelty. We would like to clarify that our work differs from a simple combination of two existing approaches, and its novelty primarily lies in two aspects: **(1) We introduce a new task: CSKG denoising, which can be applied to various CSKG construction and LLM distillation works. (2) We propose a novel approach that leverages both local and global information from textual data and graphs.**\n\nWe would like to highlight that **we are the first to propose the denoising task upon Commonsense Knowledge Graphs (CSKGs) and associate that with an effective framework**. In Section 6, we show that models in our proposed task **bring significant improvements on downstream tasks** such as Commonsense Question Answering (CSQA): denoising models in our proposed task  (i.e., our proposed method to ATOMIC10X [1]) brings a 7.0% improvement in the PIQA dataset and a 6.4% improvement in the CSQA dataset when evaluating a zero-shot QA model trained on our denoised ATOMIC-10X. Those significant improvements demonstrate that **this task holds great importance in the present context, especially with the emergence of more distillation works from large language models**. Unlike the traditional approach of human annotation and critic filtering, our task provides a distinct option for cleaning distillation data. **Our denoising framework can be applied to any CSKG without the need for additional annotation or supervision signals, making it a performant yet easy-to-use plugin for CSKGs**. Thus, our proposed task and our accompanying method serve as a novel and impactful contribution to the NLP community.\n\nWe would like to emphasize that our approach stands out by being the first to extract path-based logical rules and employ an RNN-based rule encoder to generalize them for denoising purposes. Figure 2 illustrates the joint training process, where a rule encoder in the global component compares triple and rule similarity. The resulting information propagates to the bottom-level triple encoder, enabling simultaneous parameter updates in the local component. **This aspect distinguishes our framework from a mere combination of existing modules**. Moreover, we enhance our method by incorporating semantic information derived from Large Language Models (LLMs), enabling better adaptation to commonsense knowledge scenarios. Thus, we firmly believe that our proposed approach contributes beyond simply combining two strategies, but rather organically adapting them to the task of noise detection and the domain of commonsense reasoning.\n\n# Reference\n\n[1] Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 4602– 4625. Association for Computational Linguistics.\n\n***We genuinely appreciate your input and hope that our previous responses have adequately addressed your concerns and will help you improve your scores. If you have any further questions, please don't hesitate to ask. We are more than happy to provide additional clarification and assistance.***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A good motivation to utilize local and global information jointly and successfully implement it via a simple but effective modeling method.\n\n2. The comprehensive experimental results demonstrate the effectiveness of the proposed method, especially compared to the LLM and deployed to downstream tasks.  ", "reasons_to_reject": "1. The writing, especially in the introduction, could be improved. It is difficult to fully understand the shortcomings of existing methods and the novelty of utilizing local and global information jointly.", "questions_to_authors": "I am only concerned about the novelty of the implementation of global and local information modeling. Have they been separately proposed in existing studies, and you are simply combining them, or is there a new approach to combining their strategies? Your explanation would help me better understand your innovative contributions and potentially raise my rating to a 4. ", "paper_topic_and_main_contributions": "This paper aims to detect the noisy triples in the commonsense knowledge graph.\n\nThe main contribution is that they consider both the semantic information and the global and local information jointly when detecting noise. And this method obtains a significant performance improvement.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "of0TROoNtz", "reviewer": "Reviewer_ys1w", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. In the following paragraphs, we would like to clarify your concerns one by one.\n\n# Evaluation Metric in Experiments\n\nThanks for raising the issue of evaluation metric. We would like to clarify that our usage of AUC and Recall scores is consistent with previous works’ practice [2, 3]. As demonstrated in their works, such an evaluation is resilient and reliable in terms of detection rate and quality. Actually, in our paper, the reported Recall@k is equivalent to Precision@k, as defined below. \n\n$$Recall@k = \\frac{| \\text{Noises in Top } k \\\\% |}{| \\text{Number of noises}|}$$\n\n$$Precision@k =  \\frac{|  \\text{Noises in Top } k \\\\% |}{| \\text{Number of triples} \\times k \\\\% |}$$\n\nNevertheless, we also incorporate your suggestion and included the reporting of precision and recall when the noise ratio $k\\\\%$ increases accordingly. \n\n| Method               | P@1   | R@1   | P@2   | R@2   | P@3   | R@3   | P@4   | R@4   | P@5   | R@5   |\n|----------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| GOLD (T5-xxl)        | 0.999 | 0.200 | 0.996 | 0.398 | 0.982 | 0.589 | 0.927 | 0.741 | 0.842 | 0.842 |\n| LASS (roberta-large) | 0.903 | 0.181 | 0.880 | 0.352 | 0.857 | 0.514 | 0.800 | 0.640 | 0.730 | 0.730 |\n| KG-BERT              | 0.932 | 0.187 | 0.889 | 0.356 | 0.795 | 0.487 | 0.686 | 0.551 | 0.601 | 0.601 |\n\nThe results above are obtained on the ConceptNet-N5 dataset, where the total noise injection ratio is 5%. It can be observed that our model significantly outperforms the baseline in terms of both recall and precision across these percentages.\n\nDue to the insufficient time, we only reproduce two strong baselines and our own method. We will follow your advice and incorporate a more comprehensive evaluation table in our final version.\n\n# Evaluation Details and Configuration\n\nThanks for your question. We want to emphasize that our approach builds upon previous work (line 410), where noise is introduced into the original CSKG at different ratios. This method of noise injection has been widely adopted and justified by several subsequent studies. For implementation specifics, please refer to line 308 of the supplementary material file our_readers.py. By utilizing this robust evaluation setup, we can thoroughly assess the performance of our model across various noise ratios. Furthermore, in Section 6.2 and Appendix D, we present comprehensive results from ablation studies and elaborate on the contributions of each module to noise detection. Thus, we believe that our work, along with those previous works, possesses justifiable evaluation details and setups.\n\n# Inclusion of LP Models as Baselines\n\nWe would like to clarify that our comparison with classic LP models is consistent with previous studies [1, 2, 3]. Additionally, we evaluated our method against two robust baselines [1, 3] that are specifically designed for denoising. The results demonstrate that our approach outperforms both baselines significantly. It is important to note that KGIst [2] was not included as a baseline in our work due to its requirement for node types, which is not a characteristic of the nodes in CSKG. Previous noise detection works also include the LP models as baselines for comparison, and we believe our baselines are comprehensive and competitive.\n\n# Definitions in Appendix\n\nThank you for bringing up this issue. We would like to clarify our definition of noise is for the readers' better understanding. In our work, we define noise as knowledge that contradicts normal commonsense recognition in the human world (line 247). To enhance readability, we have also included two representative examples in Figure 1, which allows readers to grasp the meaning of noise in CSKG more easily. While being concise and comprehensive, we have also provided an additional explanation of noise within specific constraints, namely Truthfulness and Reasonability. It is important to note that our main goal is to formally propose the task of noise detection and emphasize that denoising improves performance in downstream tasks. The specific categorization of noise is an additional and optional aspect. We appreciate your suggestion and will consider moving it to the body text if space permits in our camera-ready version.\n\n# Limitation and Future Works\n\nThank you for pointing out the limitation. We would like to clarify that our primary constraint lies in synthetic noise rather than human-verified noise, as mentioned in Lines 640-644. Additionally, we propose three potential approaches to address this limitation, which can serve as inspiration for future research. It is important to note that this limitation sheds light on the current constraints in our research and lays the groundwork for further investigations. Our suggested future works are not only practical but also hold value for other NLP researchers in the commonsense community. We believe that including this section will be advantageous for both our paper and the commonsense community. We appreciate your advice and will heed it by reducing the content pertaining to future challenges.\n\n# Research Questions and Main Contributions\n\nWe would like to clarify that in Section 3, our task definition provides a comprehensive explanation of the research question. Our objective is to perform denoising on the Commonsense Knowledge Graph (CSKG) by assigning a probability score to each triple, indicating the likelihood of it being noise.\n\nFurthermore, our study presents the following key contributions and significant findings:\n* We propose the task of commonsense knowledge graph noise detection. (Section 3)\n* We propose Gold, a novel denoising framework for CSKGs, which outperforms all existing methods and LLMs. (Section 6.1, 6.3)\n* Experiment results demonstrate that Gold successfully detects noises in real-world CSKGs and benefits downstream zero-shot CSQA tasks. (Section 6.4, 6.5)\n\n# The Value of Downstream Task\n\nWe want to emphasize that simply applying denoising techniques to a knowledge graph with synthetic noises is insufficient. To truly showcase the effectiveness of our denoising approach, it is crucial to validate its performance on downstream tasks. The value of the denoising method lies in its ability to enhance performance in these tasks. Our experiments on ATOMIC10X and zero-shot commonsense QA provide validation for our hypothesis, confirming the significance of our denoising approach.\n\nAs Reviewer 68ji pointed out,\n```\nThe comprehensive experimental results demonstrate the effectiveness of the proposed method, especially compared to the LLM and deployed to downstream tasks.\n```\nIn addition, our paper includes a five-page appendix that contains comprehensive experimental details and in-depth analysis. \n\n# Reference\n\n[1] Ruobing Xie, Zhiyuan Liu, Fen Lin, and Leyu Lin. 2018. Does william shakespeare REALLY write hamlet? knowledge representation learning with confidence. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4954–4961. AAAI Press.\n\n[2] Caleb Belth, Xinyi Zheng, Jilles Vreeken, and Danai Koutra. 2020. What is normal, what is strange, and what is missing in a knowledge graph: Unified characterization via inductive summarization. In WWW’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 1115–1126. ACM / IW3C2.\n\n[3] Qinggang Zhang, Junnan Dong, Keyu Duan, Xiao Huang, Yezi Liu, and Linchuan Xu. 2022b. Contrastive knowledge graph error detection. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022, pages 2590–2599. ACM.\n\n***We hope the above responses address your concerns and will help you improve your scores. Please let us know if you have other questions. We are delighted to provide additional clarification and assistance.***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work targets a hot topic in the research, given the recent advancements, that is, identifying those textual facts producing noise in data collection.\n\nThe authors model an interesting idea to detect noise in such data by composing some existing components well adapted to the specified task. In doing so, they profoundly describe the state-of-the-art on which their proposal is positioned.\n\nAlso, the work presents a precise formalization and description of the overall architecture, although some essential definitions are left in the Appendixes.", "reasons_to_reject": "The experiment run in this work can not properly state and evaluate the topic of interest treated by the authors. In fact, only the recall metrics and AUC could not completely evaluate such a system specifically designed to find noise. Specifically, precision would complete the overall model behaviour regarding noisy triples.\n\nThe evaluation task lacks relevant details to understand which configuration allows to stress the noise in the targeted dataset. Indeed, comparing classic LP models against the one proposed by the authors deceives the experiment, and the derived conclusion, validity.\n\nAlso, the Apenndixes are reserved to include related but optional work information. Instead, The authors have some central definitions, like the one regarding noisy triples and how to verify them, which is prominent in such a proposal. The Limitation section does not highlight the main limitations of this work but emphasises some future challenges (not the correct spot).\n\nThe research questions are not explicitly specified, as well as the main contributions, leaving the reader dazed about what the authors precisely want to achieve and how.\n\nThe downstream tasks bring no added value to the discussion. The authors could focus on the targeted task by proposing an in-depth and well-designed experimental configuration regarding noisy triples.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a novel denoising framework that acts over Commonsense Knowledge Graphs (CSKGs), usually characterized by spurious triples that reduce the quality of data collected in such a structure.\n\nThe introduced framework operates through semantic information, global rules and local structural information to distinguish noisy triples from real ones. Since CSKGs pair with Open KG (triples of text that define a fact), the authors describe an approach to identify the noise in the CSKG using entities' and triples' textual descriptions.\n\nThe evaluation stresses the link prediction task (LP) over two datasets through the Recall and Area Under the ROC Curve (AUC), focusing on how well such a model can detect noise. Hence, the authors test some downstream tasks to highlight the benefit of their approach.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vhMqJysfec", "reviewer": "Reviewer_UStH", "summary": "", "questions": "1. For the base model, such as RoBERTa or DeBERTa-v3, why do larger models achieve worse results?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the Reviewer for providing us with valuable feedback. We would like to clarify the Reviewer’s concerns about novelty and address the comments related to the experiments.\n\n# Novelty of Our Method\n\nThanks for raising the issue of novelty. We would like to clarify that our work goes beyond a mere combination of two existing approaches, and its novelty primarily stems from two key aspects: **(1) We introduce a new task: CSKG denoising, which can be applied to various CSKG construction and LLM distillation works. (2) We propose a novel approach that leverages both local and global information from textual data and graphs.**\n\nWe would like to highlight that **we are the first to propose the denoising task for Commonsense Knowledge Graphs (CSKGs) and develop an effective framework for it**. In Section 6, we demonstrate that our proposed task **significantly improves downstream tasks** such as Commonsense Question Answering (CSQA). Specifically, our denoising models to our proposed task  (i.e., our proposed method to ATOMIC10X [1]) brings a 7.0% improvement in the PIQA dataset and a 6.4% improvement in the CSQA dataset when evaluating a zero-shot QA model trained on our denoised ATOMIC-10X. Those substantial improvements demonstrate that **this task holds great importance in the present context, especially with the emergence of more distillation works from large language models**. Unlike the traditional approach that relies on human annotation and critic filtering, our task provides a distinct option for cleaning distillation data. **Our denoising framework can be applied to any CSKG without the need for additional annotation or supervision signals, making it a powerful yet easy-to-use plugin for CSKGs**. Thus, our proposed task and accompanying method make a novel and impactful contribution to the NLP community.\n\nWe would like to emphasize that our approach stands out by being the first to extract path-based logical rules and employ an RNN-based rule encoder to generalize them for denoising purposes. Figure 2 illustrates the joint training process, where a rule encoder in the global component compares triple and rule similarity. This information propagates to the bottom-level triple encoder, enabling simultaneous parameter updates in the local component. **This unique characteristic distinguishes our framework from a mere combination of existing modules**. Furthermore, we enhance our method by incorporating semantic information derived from Large Language Models (LLMs), which improves its adaptability to commonsense knowledge scenarios. Consequently, we firmly believe that our proposed approach goes beyond simply combining two strategies, but rather organically adapts them to the task of noise detection and the domain of commonsense reasoning.\n\n# Sample-based Analysis\nWe would like to clarify that for the manually labeled CSKG, like ConceptNet and Atomic datasets, we would assume all of the data entries are correct. However, for the LLM-distilled large-scale CSKG, such as Atomic10X [1], a CSKB distilled from GPT3, there exists a certain amount of noise. As stated in the original paper [1], the manual evaluation of the sampled 3,000 triplets from the entire ATOMIC10X dataset resulted in an error rate of 18.7%. \nIn line with your suggestion, we sample 100 triples from the top 1% with the highest anomaly probabilities in ATOMIC10X. We recruited three Ph.D. specialists in NLP, who can be considered experts in this topic, to annotate the data. The results revealed an average error rate of 55.7%. The IAA score is 77.3% calculated using pairwise agreement proportion, and Fleiss’s κ [2] is 0.54. The noise ratio extracted from the top 1% is significantly higher than the average error rate of the entire dataset, which indeed demonstrates the effectiveness of our approach. Due to time constraints, we will increase the sampling of the triples for manual evaluation in the subsequent revision.\n\n\n# Question about the Performance Deterioration of Large Models\nWe would like to emphasize that we thoroughly reviewed the implementation details and did not find any issues with the implementation. Additionally, we would like to clarify that there is no guarantee that a large-size model will outperform a base-size model, particularly when the parameters are frozen. As an example in other zero-shot settings, in Table 1 of [3], it can be observed that deberta-large performs worse than deberta-base. Similarly, in Table 2 of [4], it is evident that the GPT2 model with 762M parameters is surpassed by the GPT2 model with 345M parameters.\n\n\n\n# Reference\n[1] Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022. Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 4602– 4625. Association for Computational Linguistics.\n\n[2] Fleiss, Joseph L. \"Measuring nominal scale agreement among many raters.\" Psychological bulletin 76.5 (1971): 378.\n\n[3] Zhaowei Wang, Quyet V. Do, Hongming Zhang, Jiayao Zhang, Weiqi Wang, Tianqing Fang, Yangqiu Song, Ginny Wong, and Simon See. 2023. COLA: Contextualized Commonsense Causal Reasoning from the Causal Inference Perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5253–5271, Toronto, Canada. Association for Computational Linguistics.\n\n[4] Bosselut, Antoine, Ronan Le Bras, and Yejin Choi. \"Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering.\" Proceedings of the AAAI conference on Artificial Intelligence. Vol. 35. No. 6. 2021.\n\n\n***We genuinely appreciate your input and hope that our previous responses have adequately addressed your concerns and will help you improve your scores. Please let us know if you have other questions. We are more than happy to provide additional clarification and assistance.***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper devises a method to detect the noise in commonsense KGs, which achieves better performace than previous baselines.\n\n2. The paper is well-written and easy to follow.", "reasons_to_reject": "1. The proposed approach is the combination of representation-based and rule mining-based methods, which lacks lacks novelty to some degree.\n\n2. It would be better to investigate the noise ratio in the commonsense knwoledge graphs via sampling, since the commnonse KGs are usually constructed by experts or humans.", "questions_to_authors": "1. For the base model, such as RoBERTa or DeBERTa-v3, why do larger models achieve worse results?", "paper_topic_and_main_contributions": "The paper proposes a method named Global and Local-aware Denoising (GOLD) for the commonsense knowledge graph noise detection task. The proposed method uses the global detector to identify correct patterns, and utilizes a local detector to measure semantic similarity of triples. The proposed method achieves superior performace on several commonsen knowledge graphs, compared with previous baselines. The authors also conduct the ablation study to analyze the effect of each model component.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a denoising framework for Commonsense Knowledge Graphs (CSKGs), which often contain spurious data. Leveraging semantic information, global rules, and local structures, this framework differentiates between authentic and noisy triples. Emphasizing link prediction across two datasets using Recall and AUC metrics, the evaluations show the value of this approach in enhancing data quality, which is further illustrated through zero-shot commonsense question-answering tasks.

This paper is technically sound and well-executed; however, the impact of the results is incremental and not particularly surprising to the community.
Decision: Accept-Findings