CompleQA: Benchmarking the Impacts of Knowledge Graph Completion Methods on Question Answering
Authors: Donghan Yu
Yu Gu
Chenyan Xiong
Yiming Yang
Venue: EMNLP 2023 Findings (2023)
Keywords: knowledge graph
link prediction
question answering

Abstract: How much success in Knowledge Graph Completion (KGC) would translate into the performance enhancement in downstream tasks is an important question that has not been studied in depth. In this paper, we introduce a novel benchmark, namely CompleQA, to comprehensively assess the influence of representative KGC methods on Knowledge Graph Question Answering (KGQA), one of the most important downstream applications. This benchmark includes a knowledge graph with 3 million triplets across 5 distinct domains, coupled with over 5000 question-answering pairs and a completion dataset that is well-aligned with these questions. 
Our evaluation of four well-known KGC methods in combination with two state-of-the-art KGQA systems shows that effective KGC can significantly mitigate the impact of knowledge graph incompleteness on question-answering performance. Surprisingly, we also find that the best-performing KGC method(s) does not necessarily lead to the best QA results, underscoring the need to consider downstream applications when doing KGC.

COMPLEQA: Benchmarking the Impacts of Knowledge Graph Completion Methods on Question Answering
Donghan Yu
Yu Gu
Carnegie Mellon University dyu2@cs. cmu. edu
The Ohio State University gu. 826eosu. edu
Chenyan Xiong
Yiming Yang
Carnegie Mellon University cxêcs. cmu. edu
Carnegie Mellon University yimingêcs, cmu edu
Abstract
Completed KG
Question Answering
How much success in Knowledge Graph Com- pletion (KGC) would translate into the perfor- mance enhancement downstream tasks an important question that has not been studied depth. In this paper, we introduce a novel benchmark, namely COMPLEQA, compre- hensively assess the influence of representative KGC methods on Knowledge Graph Question Answering (KGQA), one of the most impor- tant downstream applications. This benchmark includes knowledge graph with million triplets across 5 distinct domains, coupled with over 5,000 question-ansseede pairs and com- pletion dataset that well-aligned with these questions. Our evaluation of four well-known KGC methods in combination with two state- of-the-art KGQA systems shows that effective KGC can significantly mitigate the impact of knowledge graph incompleteness on question- answering performance. Surprisingly, find that the best-performing KGC method(s) does not necessarily lead the best QA results, underscoring the need to consider downstream applications when doing KGC!.
Performance Comparison
KGC
Question Answering
Incomplete KG
We compare the performance between incom- plete and completed KGs by examining their question- answering results. Dashed arrows in the completed KG denote new triplets added by KGC.
Figure I:
The upplications of KGs span different domains and tasks. Among them, Knowledge Graph Ques- tion Answering (KGQA), designed to answer nat- ural language questions using information from KGs, one of the most important. Extensive ap- proaches have been proposed recent years (Sun et al., 2019a; Verga et al., 2021; Ye et al., 2022; Gu and Su, 2022). Despite the rapid advancement, pre- vious studies either assume the KGs are complete or handle incompleteness via specific question- answering methods (Saxena al., 2020; Liu al., 2022; Saxena et al., 2022), lacking comprehen- sive investigation into different approaches. The impact of KGC on KGQA performance remains largely unexplored.
also
1 Introduction
The inherent incompleteness of Knowledge Graphs (KGs) well-recognized problem. As such, multitude of methods have been proposed to ad- dress this issue, spanning statistical relational learn- ing (Getoor and Taskar, 2007), embedding-based models (Bordes et al., 2013; Yang et al., 2015; Sun et al., 2019h), neural-symbolic methods (Yang et al., 2017; Sadeghian et al., 2019; Qu et al., 2021),
In our study, we seek to rectify this oversight by introducing novel benchmark COMPLEQA designed to directly and holistically assess the influ- ence of KG completion on KGQA. This benchmark comprises over three million triplets and approxi- mately 400,000 entities across 5 different domains, collectively forming the KG. The corresponding QA dataset includes over 5,000 questions, featur- ing both single-hop and multi-hop questions. These questions are sorted into three generalization levels and are sourced from the GrailQA (Gu et al., 2021) dataset. For KG completion, we employ entity- centric incompleteness to align with the QA dataset,
and recent language model-based methods (Yao
al., 2019; Wang al., 2021; Saxena et al.. 2022).
However, prior studies have largely viewed KG completion as an end in itself, neglecting to investi- gate its potential impact on subsequent applications that utilize the completed KGs.
'Our code and datasct will be released at http:://ggtab.coo//lRRoss//Coono

---

where the entities in the missing triplets correspond those appearing in the questions. Importantly, different from previous studies, we actually incor- porate predicted triplets into the KG, and use this completed KG for question answering, allowing a seamless study of various KGC and KGQA meth- ods. Our investigation incorporates the study of four representative KG completion models, namely TransE (Bordes al., 2013), DistMult (Yang et al., 2015), ComplEx (Trouillon al., 2016), and Ro- tatE (Sun et al., 2019b). For the KGQA models, we employ two published leading methods DecAF (Yu et al., 2023) and Pangu (Gu et al., 2022).
Table 1: Data Statistics the QA dataset.
Question Category
Train
Valid
Test
Total
3,395
997
973
I.I.D.
366
38
Compositional
232
225
Zero-Shot
399
367
Table 2: Data Statisties the KG completion dataset. For the definition of Incompleteness, please refer Section 2.3.
Incompleteness
Train
Valid
Test
20%
1,674,405
1,807
2,864
50%
1,667 1,667,397
4,519
7,160
80%
I, 1,660,390
7,230
11,456
and business. Within this sub-sample of Freebase, we encompass 395,963 entities, 383 relations, and 3, 3,012,911 triplets. Despite the reduction scale, we posit that this knowledge graph retains a sub- stantial breadth and can accurately emulate real- world conditions and challenges. Following this, we explain the construction of the QA and comple- tion datasets derived from this KG.
Our experiments show that the incompleteness of KGs can adversely affect QA performance, and effective KG completion can help alleviate this and improve performance by up to 6.1% across vary- ing degrees of incompleteness. Furthermore, we discern that KG completion performance does not always align with downstream performance, due to the limited overlap in triplet predictions between models and the disparate influence of these triplets on KGC and QA performance. We hope find- ings stimulate more research that perceives KG completion not merely as an isolated objective but crucial step toward enhancing the performance of downstream tasks.
2.2 Question Answering over KG
We denote QA model intended for KGG as JQA and follow previous KGQA settings (Yih et al., 2016; Trouillon et al., 2016; Gu et al.. 2021), where each natural language question g aligns with an answer set A and a logical form used extract answers from the KG. We adopt GrailQA (Gu et al., 2021) as our foundation, which contains both single-hop and multi-hop questions and provides comprehensive evaluation over three generalization difficulty levels: i.i.d., composi- tional, and zero-shot, determined by whether the KG schemas that appeared in the test set have been observed in the training set. The original GrailQA is based on the full set of Freebase, thus we elim- inate questions that cannot be answered or con- tradict the provided answer labels given our sub- sampled Freebase, This refinement process results in a total of 5,365 questions with the statistical details in Table 1.
2 Benchmark Construction
In this section, we delineate the procedural steps employed in the development of our benchmark, including the knowledge graph, the question- answering dataset, and the completion dataset.
2.1 Knowledge Graph
A knowledge graph can be denoted as KCG ((E, R)C,C),T), where & is the set of entities, R is the set of relations, C is the set of classes, L is the set of literals, and T is the set of triplets. A triplet, typically denoted as (h, r),, includes the head entity h E ε, relation the respective class or literal t e (E UCU L).
R and tail entity or
2.3 KG Completion
The choice of the knowledge graph is funda- mental aspect, and in our study, we opt for Free- base (Bollacker et al., 2008), primarily owing to its widespread usage in academic circles. The origi- nal Freebase contains over 80 million entities. To facilitate affordable exploratory research, we se- lect a subset of Freebase as our final knowledge graph by specifically confining the KG to five im- portant domains: medicine, computer, food, law,
The completion task in a KG refers to, given an in- complete triplet (h, ?) or (2, 1) where its head or tail entity is missing, the model is required to predict the missing entity?.
To align it with the QA task for studying its impact, we first identify all the unique entities
Following previous conventions, we don't consider the situation of missing classes or litcrals in this study.

---

from the validation and test questions?, denoted as coald and Etest respectively. We then retrieve all triplets linked to these entities, denoted as To and Text We randomly choose a proportion P of these triplets as the final validation and test sets, Tralid and Tiest for KG completion. All remain- ing triplets, including those unlinked to those en- tities, form the training data Ttrain. We adjust P be 20%, 50%, and 80% introduce varying de- grees of incompleteness. The quantity of triplets is detailed in Table 2. Note that compared with random sampling from the entire KG, sampling triplets based on entities appeared in the questions can better align the two tasks. Furthermore, this is also seen as realistic scenario where relatively new entities in KGs often come with incomplete triplets and their related questions are both impor- tant and challenging.
3 Methods and Evaluation Metries
This section presents the methods implemented our benchmark and the evaluation metrics em- ployed to assess their effectiveness.
3.1 Question Answering
We employ two state-of-the-art methods, namely DecAF (Yu et al., 2023) and Pangu (Gu et al., 2022). For simplicity, we employ oracle entity linking to Pangu, avoiding the need to train a specially de- signed entity linker. In contrast, DecAF operates based on text retrieval without the need for en- tity linking. Due to computational limitations, we chose T5-base (Raffel et al., 2020) the backbone model for DecAF and BERT-base (Devlin et al., 2019) for Pangu. Fl scores of answer matching are used for the evaluation of QA performance.
2.4 Effect of KG Completion over KGQA
3.2 KG Completion
To study the impact of knowledge graph comple- tion on question answering, we incorporate the completed triplet into the KG, then compare the QA performance using the original incomplete KG with the completed KG. This allows us to freely explore any KGC and question-answering methods. The process of incorporating triplets is detailed below.
The KGC methods employed in this study com- prise four representative models: TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), Com- plEx (Trouillon et al., 2016), and RotatE (Sun et al.. 2019 2019b). For the implementation of these methods, we turn to LibKGE (Broscheit et al., 2020), highly developed KGC library, with details introduced in Appendix A.
In the case of each incomplete triplet (h, r, ?), the model first predicts N candidate tail entities t; with scores s; denoted as [(t1,,)). ............ (lv, sy)], where 81 2 ... 2 8N. To determine whether triplet (h, r, 4)) should be incorporated into the in- complete KG, we establish threshold ST. If the
To measure the performance of KGC, we ad- here to standard metrics for missing entity predic- tion: Mean Reciprocal Rank (MRR) and Hits @ K (H@r K) in the filtered setting (Bordes et al., 2013). Importantly, considering our goal to incorporate predicted triplets into the KG for subsequent ques- tion answering, we propose to measure triplet
score 8; 2 ST, the triplet (h, r, t4) is added to the KG. Note that we don't add the triplet which is already in the KG. The same process followed for missing head entity (?, Suppose T-red represents the collection of all added triplets for the validation set and T.prod for the test set. (T)(T) represents the KG that maintains the consistent set of entities, relations, classes, and litcrals but incorporates variable set of triplets T. Finally, we evaluate the performance of the incomplete KG (QT(())(Tiann)) versus the completed KG (aICGG(Traii U Trrred U TTered) The perfor- mance difference can indicate the utility of KG completion over the QA task. Further details on the utilized models, evaluation metrics, and how to determine sy provided in the following section.
[Tpred prediction by F1 score: F1 2 [Tpred + Tootal
where Tpred represents the set of predicted triplets while Tgold denotes the set of missing ground-trus triplets. We adjust the threshold sy introduced in Section 2.4 on the validation split to achieve the best F1 score. We show such adjustment also pro- duces good QA performance in Appendix B.
4 Experiments
In this section, we delve into the outcomes of our empirical investigations. Firstly, from Table 3, it's clear that the KG incompleteness negatively affects performance. Specifically, the F'l scores of DecAF' and Pangu drop by 14.0% and 12.1 1% respectively when the KG incompleteness level 50%. Subse- quently, we aim to address the following questions:
*GriilQA provides annotated labels for those entities.

---

Table Experiment results on KG completion and question answering our benchmark COMPLEQA. Results are averaged across 3 independent runs with different random seeds. "QA w/ X" signifies the QA performance using model X. The top performance in each column of each section is highlighted in bold. For QA performance, we provide percentage change against scenarios where no KG completion was used, which is color-coded; green for positive changes and red for negative ones.
KGC
QA DeeAF
QA Pangu
Incompleteness
Model
MRR
H
H
H@ 10
FI
0%
0.782
0.880
None
0.742
0.846
TransE
0.614
0.533
0.667
0.761
0.467
0.750 11.1%
0.864 1.21 12.1%
20%
RotatF:
0.667
0.612
0.700
0.763
0.573
0.765 13.15 13.1%
0.864 1.2.
DistMult
0.671
0.618
0.718
0.754
0.637
0.757
0.865 12.2%
ComplEx
0.681
0.630
0.722
0.756
0.660
0.757 12.0%
0,868 12.6%
None
0.672
0.773
TransF:
0.462
0.392
0.500
0.590
0.439
0.688 11.8%
0.765 11.0%
50%
RotatF:
0.509
0.455
0.538
0.607
0.524
0.701 14.3%
0.778 11.6%
DistMult
0.506
0.471
0.528
0.564
0.565
0.713 76.1%
0,798 13.2%
ComplEx
0.504
0.467
0,527
0.564
0.582
0.712 76.0%
0.798
13.2%
None
0.586
0.638
TransF
0.299
0.240
0.324
0.408
0.335
0.578 J1.4%
0.650 f11.%
80%
RotatF: 0.312
0.265
0.328
0.403
0.373
0.591 10.9%
0.636 L0.3%
DistMult 0.255
0.218
0,270
0.320
0.311
0.585 J0.2%
0.640 10.3%
ComplEx
0.252
0.212
0.272
0.323
0.320
0.588 70.3%
0.650 11.9%
Q1: How much does the good performance in
ComplEx-DecAF
ComplEx-Pangu
KGC translate to the enhancement in QA? The experimental results in Table 3 demonstrate that KGC can indeed enhance the QA performance in the majority of the cases. Notably, the ComplEx algorithm for KGC boosted the QA performance of DecAF alone (without KGC) by 6.0% and of Pangu by 3.2% at 50% incompleteness level. However, at the 80% incompleteness level, the same Com- IEx algorithm only boosted the QA performance of DecAF by 0.3% and of Pangu by 1.9%, which are much less than the performance gains at the 50% and 20% incompleteness levels. The observa- tions on other KGC methods demonstrated similar patterns; some of them even lead to decreased QA results. This could be attributed to that incorrect triplets introduced by the KGC methods outweigh the correctly predicted ones.
0.750
0.85
0.725
0.675
0.75
0.650
0.10
0,600
KGCT + 0A
0.65
KGCY
20
50 Incompletenes:%
60
20
50
80
Incompletenesst
Figure 2: QA performance under different completed KGs. "QA only" means using the original incomplete KG for QA. "KGC + QA" means using the KGC- completed KG. KGC# means only keeping the correct triplets while discarding all the incorrect ones.
although ComplEx achieves exceptional KGC sults 20% incompleteness, it does outperform RotatE in terms of QA performance when DecAF used. Similarly, even though RotatE produces top-tier KGC outcomes at 80% incompleteness, even leads to the worst QA performance when Pangu is utilized.
To validate this, we construct scenario that incorporates only the porrectly-predictedd triplets into the KG, while all the incorrect triplets are discarded. Figure 2, along with other results in Appendix C. clearly illustrates a significant perfor- mance enhancement in this scenario, especially at the 80% incompleteness level, thereby substantiat- ing the detrimental impact of incorrect predicted triplets on the QA model.
To quantitatively assess the relationship between the performance of KGC and QA, we use Spear- man't 's rank correlation coefficient. This measures the Pearson correlation between the ranked val- ues of two variables. In many instances, we find that the correlation between the KGC metric and QA metric is high, with value below 0.5. Another observation is that the Fl score of KGC performance corresponds better with QA perfor- mance than the mean reciprocal rank (MRR). This is reasonable because QA performance relies on
Q2: Does the best KGC method(s) always lead the best QA performance? According to Table 3, we see that better KGC does not always translate to better downstream outcomes. As an illustration,

---

Table 4: Spearman's 's rank correlation coefficient be- tween KGC metrics (MRR and FI) and QA metrics (FI) with two models Ac and Pangu.
potential downstream tasks. Other tasks like rec- ommendation systems and semantic search may also benefit from KGC, and their responses to KGC could differ from KGQA. Therefore, more research is needed to fully comprehend the impact of KGC on these diverse applications.
Incompleteness
Metrics
FI-DecAF
FI-Pangu
MRR
0.32
0.95
20%
0.32
0,95
MRR
0.40
0.32
50%
0.80
0.95
Additionally, our study did not employ large language models for few-shot learning in ques- tion answering, technique that is in vogue in the field of NLP. Assessing the impact of KGC in such scenarios would provide fascinating insights and help determine if our conclusions hold true in these trending methods.
MRR
0,20
-0.63
80%
F1
0.40
-0.30
the completed knowledge graph, which includes predicted triplets. The Fi score, in this context, is a more direct measure of the quality of predicted triplcts compared to MRR.
To delve further, we measured the overlap of pre- dicted triplets among various models and found that model's predictions, correct incorrect, fully encompassed the other's. For example, in 20% incomplete KG, ComplEx and RotatE had about 80% shared correct predictions and 30% shared incorrect ones. In this case, despite ComplEx's 's superior performance in KGC, it doesn't solely determine QA performance as various predicted trpletss impact QA differently, and this impact may not align well with their contribution to KGC per- formance. This discrepancy points the need for KGC methods that optimize both KG completion and downstream task performance.
Ethies Statement
An important ethical concern arising from our re- search the potential misuse of Knowledge Graphs (KGs) augmented by Knowledge Graph Comple- tion (KGC) methods. While KGC aims to enhance KG completeness and boost Question Answering (QA) systems' accuracy, it may inadvertently in- troduce or propagate biases or inaccuracies if the KGC algorithms exhibit inherent flaws.
Such biases could affect QA systems, leading skewed or misleading responses that can foster mis- information or reinforce existing biases, especially in critical decision -aainng domains. This under- lines the need for responsible use and continuous evaluation of KGC algorithms, involving rigorous validation of predicted triplets and transparency about algorithmic workings and limitations. Future work must focus devising methods to identify, measure, and rectify biases within KGC algorithms, promoting ethical use of these technologics.
5 Conclusion
this study, we introduced novel benchmark to investigate the impact of representative KGC meth- ods on the Knowledge Graph Question Answering (KGQA) task. Our findings demonstrate that KG incompleteness negatively affects KGQA, and ef- fective KGC can significantly mitigate this issue. However, also discovered that best-performing KGC method does not necessarily lead to the best KGQA results. Our work underlines the necessity to view KGC not merely as a standalone goal, but as a vital step toward improving downstream tasks.
References
Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD Conference.
Antoine Bordes, Nicolas Usunier. Alberto Gareía- Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. In Advances Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Pro- ceedings of meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States.
Limitations
While our study offers new insights into the rela- tionship between Knowledge Graph Completion (KGC) and Knowledge Graph Question Answer- ing (KGQA), it has several limitations that require further exploration.
Samuel Broscheit, Daniel Ruffinelli, Adrian Kochsiek, Patrick Betz, and Rainer Gemulla. 2020. LibKGE knowledge graph embedding library for reproducible research. In Proceedings of the 2020 Conference on
The primary focus of our study was to examine the influence KGC on KGQA, which, although an essential application KGs, is just one of many

---

Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computa- tional Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21.
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers). As- sociation for Computational Linguistics.
Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding. and Daisy Zhe Wang. 2019. DRUM: end-to- end differentiable rule mining knowledge graphs. In Advances in Neural Information Processing Sys- tems 32; Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada.
John C. Duchi, Elad Hazan, and Yoram Singer. 2010. Adaptive subgradient methods for online learning and stochastic optimization. In COLT 2010- The 23rd Conference on Learning Theory, Haifa, Israel, June 27-29, 2010. Omnipress.
Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022. Sequence-to-seque -oseeuueneeeeeeeeeeeeeeeee knowledge graph com- pletion and question answering. In Proceedings of the 60th Anmual Meeting of the Association for Com- putcational Linguistics (Volume 1: Long Papers). As- sociation for Computational Linguisties.
Lise Getoor and Ben Taskar. 2007. Introduction to statistical relational learning. MIT press.
Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics. Association for Computational Linguistics.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural net- works. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings.
Haitian Sun, Tania Bedrax- and William Cohen. 2019a. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage P Processing (EMNLP-JJCNLP) Association for Computational Linguisties.
Yu Gu, Xiang Deng, and Yu Su. 2022. Don't generate, discriminate: A proposal for grounding language models to real-world environments. ArXiv preprint, abs/2212.09736..
Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond iid: three levels of generalization for question answering on knowledge bases. In Proceedings of the Web Conference 2021.
Zhiqing Sun. Zhi-Hong Deng, Jian- Yun Nie, and Jian Tang. 2019b. Rotate: Knowledge graph embedding by relational rotation in complex space. In 7th In- ternational Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Yu Gu and Yu Su. 2022. ArcaneQA: Dynamic program induction and contextualized encoding for knowl- edge base question answering. In Proceedings of the 29th International Conference Computational Lin- guistics. International Committee on Computational Linguistics.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceed- ings of the 33nd International Conference on Ma- chine Learning, ICMI. 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings. JMLR.org.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William Cohen. 2021. Adaptable and interpretable neural MemoryOver symbolic knowledge. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies. Associ- ation for Computational Linguistics.
Lihui Liu, Boxin Du, Jiejun Xu, Yinglong Xia, and
Hanghang Tong. 2022. Joint knowledge graph com- pletion and question answering. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining.
Meng Qu, Junkun Chen, Louis-Pascas A. C. Xhonneux, Yoshua Bengio, and Jian Tang. 2021. Rnnlogic: Learning logic rules for reasoning on knowledge graphs. In 9th International Conference on Learning Representations, ICLR 2021. Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Xiaozhi Wang. Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. KEPLER: A unified model for knowledge embed- ding and pre-trained language representation. Trans- actions of the Association for Computational Linguis- tics, 9.

---

Bishan Yang, Wen-tau Yih. Xiaodong He, Jianfeng Gao. and Li Deng. 2013. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Represen- lations, ICLR 2015, San Diego. CA, USA, May 7-9. 2015, Conference Track Proceedings.
A Knowledge Graph Completion
We introduce the KG completion models based on the LibKGE (Broscheit et al., 2020) library.
settings for our
The dimensionality of our embeddings is set to 200, utilizing xavier uniform initialization (Glorot and Bengio, 2010). Following previous approaches (Bordes al., 2013; Sun et al., 2019b), we em- ploy negative sampling for training. In this process, negative triplets are created for every triplet in the training dataset by randomly substituting either the head or tail entity. Each triplet gives rise to 50 neg- ative triplets, half by replacing the head entity and half by substituting the tail. The batch size is 4096, with maximum of 500 epochs. We deploy early stopping after 5 evaluations over the validation set, with one evaluation after every 10 epochs.
Fan Yang. Zhilin Yang, and William W. Cohen. 2017. Differentiable learning of logical rules for knowledge base reasoning. In Advances in Neural Information Processing Systems 30: Annual Conference Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kg- bert: Bert for knowledge graph completion. ArXiv preprint, htss/999..0193.
Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2022. RNG-KBQA: Genera- tion augmented iterative ranking for knowledge base question answering. Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Association for Computational Linguistics.
When comes to optimization, we leverage Adam (Kingma and Ba, 2015) for TransE (Bor- des et al., 2013) and RotatE (Sun et al., 2019b) with a learning rate of 0.001. For ComplEx (Trouil- lon et al., 2016) and DistMult (Yang et al., 2015), employ Adagrad (Duchi al., 2010) with learning rate within the range [0.01, 1.0].
Wen-tau Yih, Matthew Richardson, Chris Meek, Ming- Wei Chang, and Jina Suh. 2016. The value of se- mantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computa- tional Linguistics.
For RotatE and TransE, we choose from binary cross entropy (BCE), Kullback-Leibler divergence (KL), and margin ranking (MR) for the training losses, and explore the loss argument within [0, 50]. For ComplEx and DistMult, we utilize BCE loss with loss argument of 0.0. We experimented with other training losses and arguments, but found them ineffective. Hyperparameter search is conducted via Bayesian optimization using the Ax framework (https://ax.dev/) with the number of trials as 40.
Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Yang Wang. Zhiguo Wang, and Bing Xiang. 2023. DecAF: Joint decoding answers and log- ical forms for question answering over knowledge bases. In The Eleventh International Conference on Learning Representations.
B Score Threshold
We aimed to study the effect of score threshold st for each KGC method, which is introduced in Sec- tion 2.4. We vary this threshold for each method to evaluate the corresponding KGC and QA perfor- mance. As shown in Figure 3, the KGC Fl score initially rises and then falls, as increasing the score threshold leads to fewer triplets being added, which although improves precision, it negatively impacts recall. We see that the curve representing the rela- tionship between the score threshold and the KGC Fl score appears to coincide well with the QA performance curve. However, there are instances where this alignment less pronounced, indicating that utilizing KGC Fl to pinpoint the task- specific score threshold provides a useful starting point, but

---

ComplEx
DistMult
0.70
0.70
0.65
0.65
0.60
QA
0.80
w/ DeCAF
0.55
0.50
0.45
0.45
1.9
2,9
30
4.9
5.9
1.5
2.5
3.5
4.5
5.5
score threshold Sy
score threshold
TransE
0.65
KGC
0.60
KGC
wi Bec
0.55
w/ DecAF
0.50
0.45
12.7 11.7
10.7
0.7
8.7
-1.88
-16.8
-15.8
16.8 11..0
score threshold Sy
score threshold Sy
DistMult-DecAF
DistMult-Pangu
0.85
Figure Performance variation with different score threshold. Each plot depicts the performance trends for one KGC method at 50% KG incompleteness level.
0.725
0.75
0.500
0.625
The curves represent changes KGC Fl scores on the validation split and QA w/ DecAF F1 scores the test split as the score threshold varies.
QA
0.65
20
50
10
20
50
80
Incompletenes>%
Incompletenessts
TransE-Decke
TransE-Pangu
is not sufficient on its own. We believe that further investigation of this issue could provide valuable insights for future research.
0,700
0.80
0.675
QA only
QAlorly
0A
0,600
Filtering Incorrect Triplets
0.575
0.05
50
50
Incompleteness%
Incompleteness%
As additional experiments for Section 4.1, we present the question-answering performance of other KGC methods besides ComplEx after dis- carding incorrect triplets and retaining only the correct ones. As depicted in Figure it's clear that QA efficacy significantly improves when incorrect triplets are excluded, notably when the knowledge graph is 80% incomplete, which reflects a situation where KGC quality is notably poor, with high frequency of incorrect triplet predictions.
RotatE-DecAF
RotatE-Pangu
0.758
0.85
0.725
0.75
0.625
0.356
GA only
KGC QA
0.70
D2A only
KCC I
KGC
Incompleteness%
Incompleteness%
Figure 4: QA performance under different completed KGs. "QA only" means using the original incomplete KG for QA. "KGC + QA" means using the KGC- completed KG. KGC means only keeping the correct triplets while discarding all the incorrect ones.


Reviews:
[{"review_id": "2abDelRfFD", "reviewer": "Reviewer_dwAq", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting research area\n- Combination of results from different domains (QA, embedddings)\n", "reasons_to_reject": "- No major scientific contribution\n- No surprising insights\n- Results achieved on one dataset", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a benchmark/benchmarking approach to evalaute the impact of predictions made by knowledge graph embeddings on question answering. The benchmark datasets consists of 3M triples with 5K questions. The introduction references a large number of KGQA approaches and corresponding benchmarking. The authors then delve into the construction of their benchmark. In particular, the KG completion is carried out using embedding methods. The evaluation with two state-of-the-art approaches suggests that increasing the incompleteness leads to worse results. Adding only correct triples tends to improve the results but in some cases, adding predicted triples actually worsens the performance. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bZhVg943r6", "reviewer": "Reviewer_kWfn", "summary": "", "questions": "1) Can you give more intuition behind the choice of Pangu and DecAF as proposed KGQA baselines? Is there a reason that a semantic parsing technique was not considered? I accept the LLM explanation in Limitations but am skeptical at the lack of any representation of semantic parsing techniques for this task. \n\n2) Did you consider weighting obscured triples by predicate or some other strategy to control the rate that KGC models get credit for completing irrelevant triples? \n\n3) Is it possible to include examples of incomplete triples that were filled and subsequently resulted in a correct output? I.e. do you have/can you include some examples where KGQA is incorrect because of incompleteness and then made correct by KGC? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A strength of this paper is its focus on grounding a long studied academic Graph task (KGC) in a long standing downstream NLP task (KGQA). The intuition to ground a task in its benefits to other tasks gives weight to the importance of their proposal on application in this domain. Additionally, the authors start from a widely cited and popular benchmark (GrailQA) that includes questions from IID, compositional, and zero-shot domains; this adds empirical weight to their findings because this is widely thought to be a high quality benchmark. Lastly, the authors select a representative set of popular KGC methods. In terms of writing, this paper is well written and easy to follow. ", "reasons_to_reject": "I think there are two reasons to consider rejecting this paper:\n1) The notion of incompleteness for their KGC task is somewhat artificial; the authors obtain relations of answer entities in dev/test set, and  \"incomplete\" them by obscuring one of the entities in the relation, and then \"randomly choose a proportion P of these triplets as the final validation and test sets\". This challenge here is that not all triples of answer entities are equally meaningful for QA; i.e. it is possible that some and probably likely that some, if not most, triples, are largely meaningless. As a result, obscuring them would be expected to have little/no impact on QA. This risks significantly diluting the correlation between KGC quality and downstream QA. Specifically, a KGC model might look great because it fills in lots of relations but then look bad for QA because some large percent of those relations were meaningless. This can potentially mislead the reader into thinking that a particular KGC model is strong on KGC but bad for QA, when in fact the measurement of KGC model strength on this benchmark may just be an artifact of the benchmark. I think this can be improved by applying stratified sampling over predicates to ensure that we are controlling not just for the entities we obscure but also for relative importance of the information the KGC model completes. Further motivating the need for a deeper look into this approach is Appendix D, which seems to indicate a weak correlation between KGC and KGQA. It would be interesting to see the set of questions for which there is little to no correlation, as this is where the most opportunity is to improve this benchmark. Ultimately I view this as something that dilutes, but does not invalidate, the conclusions. There is a correlation, and that correlation needs to be explored further, but unfortunately the authors don't dig into the link between KGC and KGQA much to explain this correlation.\n\n2) Modeling techniques studied; the authors study two KGQA techniques, Pangu and DecAF. While both exhibit strong performance, these are not the most natural nor most representative baseline choices. Neither of these techniques are popular or widely cited; DecAF is somewhat popular and adopts an IR approach to KGQA, in which documents of triples are retrieved as evidence to inform answer generation, while Pangu is not and employs a neurosymbolic planner/critic pipeline. This paper would be made more sound by the inclusion of more popular and representative techniques, especially those of the semantic parsing variety, of which there are many popular approaches (citations below). The lack of semantic parsing baseline is concerning for a few reasons, including that 1) these are the most widely studied approach to KGQA and 2) semantic parsing provides a more direct means to assess KGC. W/r/t 2) specifically, these approaches would enable the authors to measure the benefit of KGC on generated parse success rate. For example, using a parsing approach trained on the complete KG and evaluated on the incomplete one would provide a realistic measurement of the impact of missing data, as parses that seek to return the obscured data would no longer be executable. The authors should give more justification behind why they chose DecAF and Pangu, beyond just \"here are 2 SOTA techniques\". While they discuss the lack of an LLM-only baseline in the limitation, they don't include semantic parsing. \n\n@article{Ye2021RNGKBQAGA,\n  title={RNG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering},\n  author={Xi Ye and Semih Yavuz and Kazuma Hashimoto and Yingbo Zhou and Caiming Xiong},\n  journal={ArXiv},\n  year={2021},\n  volume={abs/2109.08678},\n  url={https://api.semanticscholar.org/CorpusID:237562927}\n}\n\n@inproceedings{Abdelaziz2021ASP,\n  title={A Semantic Parsing and Reasoning-Based Approach to Knowledge Base Question Answering},\n  author={I. Abdelaziz and Srinivas Ravishankar and Pavan Kapanipathi and Salim Roukos and Alexander G. Gray},\n  booktitle={AAAI Conference on Artificial Intelligence},\n  year={2021},\n  url={https://api.semanticscholar.org/CorpusID:235363625}\n}", "questions_to_authors": "1) Can you give more intuition behind the choice of Pangu and DecAF as proposed KGQA baselines? Is there a reason that a semantic parsing technique was not considered? I accept the LLM explanation in Limitations but am skeptical at the lack of any representation of semantic parsing techniques for this task. \n\n2) Did you consider weighting obscured triples by predicate or some other strategy to control the rate that KGC models get credit for completing irrelevant triples? \n\n3) Is it possible to include examples of incomplete triples that were filled and subsequently resulted in a correct output? I.e. do you have/can you include some examples where KGQA is incorrect because of incompleteness and then made correct by KGC? ", "paper_topic_and_main_contributions": "Paper is about KG completion (KGC) and impact on downstream question answering. It aims to motivate the KGC problem by showing that KGC impacts QA performance, which is a significant application of KGs. Contributions are NLP engineering experiment and data resource derived from existing artifacts (GrailQA, Freebase). Specifically:\n1) Argumentation for the important of grounding the KGC task in its impact on downstream performance\n2) Benchmark for measuring KGC task benefit on KGQA using a small subset of Freebase (400K entities out of 80M) and questions mined from GrailQA\n3) Empirical comparison of popular KGC techniques on 2 KGQA techniques (Pangu, DecAF). \n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "048: Nit: \"operate on complete KGs\" would benefit from citation or further clarity on what you mean. Popular KGs are Freebase, Wikidata, Yago, and DBPedia, but I am not sure that any of these are considered complete. I think you mean \"prior approaches ignore completeness and assume KB is complete or specifically design solutions to perform without completing KB\"\n\nTable 1 and 2: Nit: i think it could help to add a caption to these. They are presented before the paragraphs that explain them so I am not sure what \"incompleteness\" percent in Table 2 specifically means until I read the rest of the paper. \n\nAppendix D: feels important and I think it would improve the paper to move this into the discussion. "}}, {"review_id": "ADvxxb4nNW", "reviewer": "Reviewer_TkCf", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well written and motivated, and it is easy to follow.\n* The experimental design is sound and detailed.\n* Interesting and novel task that has been overlooked in the literature. Has potential on further defining the practical usefulness of KGC by influencing the way of evaluating of KGC in the context of practical applications.", "reasons_to_reject": "The paper lacks analysis that would give insights on why better KGC performance does not align with better KGQA performance. This includes quantitative analysis, e.g,. the entity types / relations for which this phenomenon is more frequent, and qualitative analysis, e.g., questions that are anyway harder to answer even if the correct triples have been predicted by KGC. Since this is a short paper, this could be included in the Appendix.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a benchmark that investigates the influence of knowledge graph completion (KGC) on knowledge graph question answering (KGQA), as a prominent application. The paper builds on existing datasets to create a dataset specifically for this benchmark. The experiments test the robustness of use state-of-the-art KGC and their influence on KGQA methods. The paper suggests that effective KGC can improve KGQA but often better KGC performance does not align with better KGQA performance.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KOmEgITkwH", "reviewer": "Reviewer_81Bu", "summary": "", "questions": "The correlation Between KGC and QA is a good result. But how do you propose to solve this rather than pure observation?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The experimental results seem solid.\nThe writing is clear and easy to follow\nThe question is interesting and worth investigation", "reasons_to_reject": "1. The overall setting is weird to me. The incompletion ratio is set to 20, 50, 80%. however, how would one opt to such a KG system with such an extremely high incompletion ratio? This is not realistic.\n2. The incompletion in KG can sometimes lead to improvement in QA. This is an interesting finding, but how would you support the QA system with this finding? I did not see a deeper analysis of leveraging this and how to overcome the drawbacks.", "questions_to_authors": "The correlation Between KGC and QA is a good result. But how do you propose to solve this rather than pure observation?", "paper_topic_and_main_contributions": "This paper investigates how the incompletion of KG influence downstream QA task. They found incompletion usually leads to worse QA performance but sometimes improves it.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary:**
The paper introduces a benchmarking approach to assess the impact of predictions generated by knowledge graph embeddings on question answering. The benchmark dataset comprises 3 million triples and 5,000 questions. The experiments aim to evaluate the robustness of state-of-the-art knowledge graph completion (KGC) methods and their influence on knowledge graph question answering (KGQA) techniques.
The authors provide insights into the construction of their benchmark, emphasizing the use of embedding methods for knowledge graph completion. The goal is to highlight the relevance of the KGC problem by demonstrating its impact on question answering performance, which is a significant application of knowledge graphs. The contributions of this work include NLP engineering experiments and a data resource derived from existing artifacts such as GrailQA and Freebase.
The evaluation results suggest that increasing the incompleteness of the knowledge graph leads to worse performance. While the addition of correct triples tends to improve results, there are cases where the inclusion of predicted triples actually worsens performance.

**Strengths:**
A notable strength of this paper lies in its approach to bridging a well-established academic task, knowledge graph completion, with a longstanding practical NLP task, namely, the impact of knowledge graph embeddings on question answering. 
The authors' insight into connecting a task with its potential benefits for other tasks underscores the significance of their proposal within this domain. 
Furthermore, the authors build upon a highly regarded benchmark, GrailQA, which encompasses questions from various domains, including IID, compositional, and zero-shot, lending empirical robustness to their findings, as GrailQA is widely recognized as a high-quality benchmark. Additionally, the authors carefully select a representative set of popular knowledge graph completion (KGC) methods and techniques, including Pangu and DecAF.

**Weaknesses:**
Reviewers have identified the following weaknesses in this work:
1. The concept of "incompleteness" in the knowledge graph completion (KGC) task appears somewhat artificial. The authors create this sense of incompleteness by obtaining relations of answer entities in the dev/test set and then obscuring one of the entities in the relation. Subsequently, they randomly select a proportion (P) of these triplets as the final validation and test sets. This approach might inadvertently lead readers to conclude that a specific KGC model has good performance in KGC but performs poorly in QA, when in reality, the observed KGC model strength on this benchmark may be an artifact of the benchmark construction.
2. While there is a correlation between KGC and knowledge graph question answering (KGQA), this correlation remains largely unexplored. The paper lacks a thorough analysis that would provide insights into why superior KGC performance does not necessarily translate to better KGQA performance. This analysis could encompass quantitative factors, such as entity types and relations more prone to this phenomenon, as well as qualitative aspects, like questions that remain challenging to answer even if the correct triples are predicted by KGC.
3. The paper could benefit from the inclusion of more popular and representative techniques, particularly those related to semantic parsing. The absence of a semantic parsing baseline is a concern for several reasons, including the widespread use of semantic parsing in KGQA and its suitability for assessing KGC. Incorporating such approaches would enable the authors to measure the impact of KGC on generated parse success rates.
4. The selection of DecAF and Pangu as techniques for evaluation should be substantiated further. The authors should provide more justification beyond simply stating that these are two state-of-the-art techniques. Additionally, while the limitation section mentions the absence of an LLM-only baseline, the paper does not incorporate semantic parsing, which is a notable omission.
5. The overall experimental setting, particularly the choice of incompletion ratios at 20%, 50%, and 80%, might not align with a realistic KG system with such an extremely high incompletion ratio. Further context or justification for these ratios would enhance the paper's credibility.

**Author-Reviewer discussion and acknowledgment:**
The authors have not provided any rebuttal comments.

**Conclusion:**
This paper is well-written and easy to follow. The contribution is interesting, addressing a task that has been overlooked in the literature. However, reviewers recommend that the authors incorporate more state-of-the-art research and consider additional improvements. It is suggested that the work should include additional data based on the points highlighted within the reviews. Additionally, the authors should address the identified typos in the paper.
Decision: Accept-Findings