ICML 2025

Is Complex Query Answering Really Complex?
Cosimo Gregucci 1 Bo Xiong 2 Daniel Hern´andez 1 Lorenzo Loconte 3
Pasquale Minervini 3 4
Steffen Staab 1 5
Antonio Vergari 3
Abstract
Complex query answering (CQA) on knowledge
graphs (KGs) is gaining momentum as a challeng-
ing reasoning task. In this paper, we show that
the current benchmarks for CQA might not be as
complex as we think, as the way they are built dis-
torts our perception of progress in this field. For
example, we find that in these benchmarks, most
queries (up to 98% for some query types) can be
reduced to simpler problems, e.g. link prediction,
where only one link needs to be predicted.The
performance of state-of-the-art CQA models de-
creases significantly when such models are eval-
uated on queries that cannot be reduced to easier
types. Thus, we propose a set of more challeng-
ing benchmarks composed of queries that require
models to reason over multiple hops and better
reflect the construction of real-world KGs. In a
systematic empirical investigation, the new bench-
marks show that current methods leave much to
be desired from current CQA methods.
1. Introduction
A crucial challenge in AI and ML is learning to perform
complex reasoning, i.e., solving tasks that involve several
intermediate steps and sub-goals to be completed. Complex
query answering (CQA; Hamilton et al., 2018; Zhang et al.,
2021; Arakelyan et al., 2021; Zhu et al., 2022) emerged as a
way to measure complex reasoning over external knowledge
bases, encoded as knowledge graphs (KGs; Hogan et al.,
2021). For instance, to answer the query:
“Which actor performed in a movie filmed in
New York City and distributed on Blue Ray?”
(q1)
Shared supervision 1Institute for Artificial Intelligence, Uni-
versity of Stuttgart, Germany 2Stanford University 3School of
Informatics University of Edinburgh Edinburgh, UK 4Miniml.AI
5University of Southampton, UK. Correspondence to: Cosimo
Gregucci <cosimo.gregucci@ki.uni-stuttgart.de>.
Proceedings of th...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000043. Conference bucket: ICML.