SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Erik Arakelyan*1 Pasquale Minervini*†2 Daniel Daza3,4,5
Michael Cochez<sup>3,5</sup> Isabelle Augenstein<sup>1</sup>
<sup>1</sup>University of Copenhagen <sup>2</sup>University of Edinburgh <sup>3</sup>Vrije Universiteit Amsterdam
$^{4}$ University of Amsterdam  $^{5}$ Discovery Lab, Elsevier, The Netherlands
{erik.a,augenstein}@di.ku.dk p.minervini@ed.ac.uk {d.dazacruz,m.cochaiez}@vu.nl
Answering complex queries on incomplete knowledge graphs is a challenging task where a model needs to answer complex logical queries in the presence of missing knowledge. Prior work in the literature has proposed to address this problem by designing architectures trained end-to-end for the complex query answering task with a reasoning process that is hard to interpret while requiring data and resource-intensive training. Other lines of research have proposed re-using simple neural link predictors to answer complex queries, reducing the amount of training data by orders of magnitude while providing interpretable answers. The neural link predictor used in such approaches is not explicitly optimised for the complex query answering task, implying that its scores are not calibrated to interact together. We propose to address these problems via  $\mathrm{CQD}^A$ , a parameter-efficient score adaptation model optimised to re-calibrate neural link prediction scores for the complex query answering task. While the neural link predictor is frozen, the adaptation component – which only increases the number of model parameters by  $0.03\%$  – is trained on the downstream complex query answering task. Furthermore, the calibration component enables us to support reasoning over queries that include atomic negations, which was previously impossible with link predictors. In our experiments,  $\mathrm{CQD}^A$  produces significantly more accurate results than current state-of-the-art methods, improving from 34.4 to 35.1 Mean Reciprocal Rank values averaged across all datasets and query types while using  $\leq 30\%$  of the available training query types. We further show that  $\mathrm{CQD}^A$  is data-efficient, achieving competitive results with only  $1\%$  of the complex training queries, and robust in out-of-domain evaluations. Source code and datasets are available at https://github.com/EdinburghNLP/adaptive-cqd.
A Knowledge Graph (KG) is a knowledge base representing the relationships between entities in a relational graph structure. The flexibility of this knowledge representation formalism allows KGs to be widely used in various domains. Examples of KGs include general-purpose knowledge bases such as Wikidata [Vrandecic and Krötzsch, 2014], DBpedia [Auer et al., 2007], Freebase [Bollacker et al., 2008], and YAGO [Suchanek et al., 2007]; application-driven graphs such as the Google Knowledge Graph, Microsoft's Bing Knowledge Graph, and Facebook's Social Graph [Noy et al., 2019]; and domain-specific ones such as SNOMED CT [Bodenreider et al., 2018], MeSH [Lipscomb, 2000], and Hetionet [Himmelstein et al., 2017] for life sciences; and WordNet [Miller, 1992] for linguistics.
*Equal contribution, alphabetical order. †Senior author.
Answering complex queries over Knowledge Graphs involves a logical reasoning process where a conclusion should be inferred from the available knowledge.
Neural link predictors [Nickel et al., 2016] tackle the problem of identifying missing edges in large KGs. However, in many domains, it is a challenge to develop techniques for answering complex queries involving multiple and potentially unobserved edges, entities, and variables rather than just single edges.
Prior work proposed to address this problem using specialised neural networks trained end-to-end for the query answering task [Hamilton et al., 2018, Daza and Cochez, 2020, Ren et al., 2020, Ren and Leskovec, 2020, Zhu et al., 2022], which offer little interpretability and require training with large and diverse datasets of query-answer pairs. These methods stand in contrast with Complex Query Decomposition [CQD, Arakelyan et al., 2021, Minervini et al., 2022], which showed that it is sufficient to re-use a simple link prediction model to answer complex queries, thus reducing the amount of training data required by orders of magnitude while allowing the possibility to explain intermediate answers. While effective, CQD does not support negations, and fundamentally, it relies
on a link predictor whose scores are not necessarily calibrated for the complex query answering task. Adapting a neural link predictor for the query answering task while maintaining the data and parameter efficiency of CQD, as well as its interpretable nature, is the open challenge we take on in this paper.
We propose  $\mathrm{CQD}^A$ , a lightweight adaptation model trained to calibrate link prediction scores, using complex query answering as the optimisation objective. We define the adaptation function as an affine transformation of the original score with a few learnable parameters. The low parameter count and the fact that the adaptation function is independent of the query structure allow us to maintain the efficiency properties of CQD. Besides, the calibration enables a natural extension of CQD to queries with atomic negations.
An evaluation of  $\mathrm{CQD}^{\mathcal{A}}$  on three benchmark datasets for complex query answering shows an increase from 34.4 to 35.1 MRR over the current state-of-the-art averaged across all datasets while using  $\leq 30\%$  of the available training query types. In ablation experiments, we show that the method is data-efficient; it achieves results comparable to the state-of-the-art while using only  $1\%$  of the complex queries. Our experiments reveal that  $\mathrm{CQD}^{\mathcal{A}}$  can generalise across unseen query types while using only  $1\%$  of the instances from a single complex query type during training.
Link Predictors in Knowledge Graphs Reasoning over KGs with missing nodes has been widely explored throughout the last few years. One can approach the task using latent feature models, such as neural link predictors [Bordes et al., 2013, Trouillon et al., 2016, Yang et al., 2014, Dettmers et al., 2018, Sun et al., 2019, Balažević et al., 2019, Amin et al., 2020] which learn continuous representations for the entities and relation types in the graph and can answer atomic queries over incomplete KGs. Other research lines tackle the link prediction problem through graph feature models [Xiong et al., 2017, Das et al., 2017, Hildebrandt et al., 2020, Yang et al., 2017, Sadeghian et al., 2019], and Graph Neural Networks [GNNs, Schlichtkrull et al., 2018, Vashishth et al., 2019, Teru et al., 2020].
Complex Query Answering Complex queries over knowledge graphs can be formalised by extending one-hop atomic queries with First Order Logic (FOL) operators, such as the existential quantifier  $(\exists)$ , conjunctions  $(\land)$ , disjunctions  $(\lor)$  and negations  $(\neg)$ . These FOL constructs can be represented as directed acyclic graphs, which are used by embedding-based methods that represent the queries
Figure 1: Given a complex query  $\mathcal{Q}$ ,  $\mathrm{CQD}^{\mathcal{A}}$  adapts the neural link prediction scores for the sub-queries to improve the interactions between them.
using geometric objects [Ren et al., 2020, Hamilton et al., 2018] or probabilistic distributions [Ren and Leskovec, 2020, Zhang et al., 2021, Choudhary et al., 2021] and search the embedding space for the answer set. It is also possible to enhance the properties of the embedding space using GNNs and Fuzzy Logic [Zhu et al., 2022, Chen et al., 2022]. A recent survey [Ren et al., 2023] provides a broad overview of different approaches. Recent work [Daza and Cochez, 2020, Hamilton et al., 2018, Ren and Leskovec, 2020] suggests that such methods require a large dataset with millions of diverse queries during the training, and it can be hard to explain their predictions.
Our work is closely related to CQD [Arakelyan et al., 2021, Minervini et al., 2022], which uses a pre-trained neural link predictor along with fuzzy logical t-norms and t-conorms for complex query answering. A core limitation of CQD is that the pre-trained neural link predictor produces scores not calibrated to interact during the complex query-answering process. This implies that the final scores of the model are highly dependent on the choice of the particular t-(co)norm aggregation functions, which, in turn, leads to discrepancies within the intermediate reasoning process and final predictions. As a side effect, the lack of calibration also means that the equivalent of logical negation in fuzzy logic does not work as expected.
With  $\mathrm{CQD}^{\mathcal{A}}$ , we propose a solution to these limitations by introducing a scalable adaptation function that calibrates link prediction scores for query answering. Furthermore, we extend the formulation of CQD to support a broader class of FOL queries, such as queries with atomic negation.
3 Background
A Knowledge Graph  $\mathcal{G} \subseteq \mathcal{E} \times \mathcal{R} \times \mathcal{E}$  can be defined as a set of subject-predicate-object  $\langle s, p, o \rangle$  triples, where each triple encodes a relationship of type  $p \in \mathcal{R}$  between the subject  $s \in \mathcal{E}$  and the object  $o \in \mathcal{E}$  of the triple, where  $\mathcal{E}$  and  $\mathcal{R}$  denote the set of all entities and relation types, respectively. A Knowledge Graph can be represented as a First-Order Logic Knowledge Base, where each triple  $\langle s, p, o \rangle$  denotes an atomic formula  $p(s, o)$ , with  $p \in \mathcal{R}$  a binary predicate and  $s, o \in \mathcal{E}$  its arguments.
First-Order Logical Queries We are concerned with answering logical queries over incomplete knowledge graphs. We consider queries that use existential quantification (∃) and conjunction (∧) operations. Furthermore, we include disjunctions (∨) and atomic negations (¬). We follow Ren et al. [2020] by transforming a logical query into Disjunctive Normal Form [DNF, Davey and Priestley, 2002], i.e. a disjunction of conjunctive queries, along with the subsequent extension with atomic negations in [Ren and Leskovec, 2020]. We denote such queries as follows:
$$
\mathcal {Q} [ A ] \triangleq ? A: \exists V _ {1}, \dots , V _ {m}. \left(e _ {1} ^ {1} \wedge \dots \wedge e _ {n _ {1}} ^ {1}\right) \vee \dots \vee \left(e _ {1} ^ {d} \wedge \dots \wedge e _ {n _ {d}} ^ {d}\right),
$$
where  $e_i^j = p(c,V)$ , with  $V\in \{A,V_1,\ldots ,V_m\}$ ,  $c\in \mathcal{E},p\in \mathcal{R}$  (1)
or  $e_i^j = p(V,V')$ , with  $V,V'\in \{A,V_1,\ldots ,V_m\}, V\neq V',p\in \mathcal{R}$ .
In Equation (1), the variable  $A$  is the target of the query,  $V_{1},\ldots ,V_{m}$  denote the bound variable nodes, while  $c\in \mathcal{E}$  represent the input anchor nodes, which correspond to known entities in the query. Each  $e_i$  denotes a logical atom, with either one  $(p(c,V))$  or two variables  $(p(V,V'))$ .
The goal of answering the logical query  $\mathcal{Q}$  consists in finding the answer set  $\llbracket \mathcal{Q} \rrbracket \subseteq \mathcal{E}$  such that  $a \in \llbracket \mathcal{Q} \rrbracket$  iff  $\mathcal{Q}[a]$  holds true. As illustrated in Figure 1, the dependency graph of a conjunctive query  $\mathcal{Q}$  is a graph where nodes correspond to variable or non-variable atom arguments in  $\mathcal{Q}$  and edges correspond to atom predicates. We follow Hamilton et al. [2018] and focus on queries whose dependency graph is a directed acyclic graph, where anchor entities correspond to source nodes, and the query target  $A$  is the unique sink node.
Example 3.1 (Complex Query). Consider the question "Which people are German and produced the music for the film Constantine?" It can be formalised as a complex query  $\mathcal{Q} \equiv ?T$ : country(Germany,  $T$ ) ∧ producerOf(Constantine,  $T$ ), where Germany and Constantine are anchor nodes, and  $T$  is the target of the query, as presented in Figure 1. The answer  $[\mathcal{Q}]$  corresponds to all the entities in the knowledge graph that are German composers for the film Constantine.
Continuous Query Decomposition CQD is a framework for answering EPFO logical queries in the presence of missing edges [Arakelyan et al., 2021, Minervini et al., 2022]. Given a query  $\mathcal{Q}$ , CQD
defines the score of a target node  $a \in \mathcal{E}$  as a candidate answer for a query as a function of the score of all atomic queries in  $\mathcal{Q}$ , given a variable-to-entity substitution for all variables in  $\mathcal{Q}$ .
Each variable is mapped to an embedding vector that can either correspond to an entity  $c \in \mathcal{E}$  or to a virtual entity. The score of each of the query atoms is determined individually using a neural link predictor [Nickel et al., 2016]. Then, the score of the query with respect to a given candidate answer  $\mathcal{Q}[a]$  is computed by aggregating all of the atom scores using t-norms and t-conorms – continuous relaxations of the logical conjunction and disjunction operators.
Neural Link Predictors A neural link predictor is a differentiable model where atom arguments are first mapped into a  $d$ -dimensional embedding space and then used to produce a score for the atom. More formally, given a query atom  $p(s,o)$ , where  $p \in \mathcal{R}$  and  $s,o \in \mathcal{E}$ , the score for  $p(s,o)$  is computed as  $\phi_p(\mathbf{e}_s,\mathbf{e}_o)$ , where  $\mathbf{e}_s,\mathbf{e}_o \in \mathbb{R}^d$  are the embedding vectors of  $s$  and  $o$ , and  $\phi_p: \mathbb{R}^d \times \mathbb{R}^d \mapsto [0,1]$  is a scoring function computing the likelihood that entities  $s$  and  $o$  are related by the relationship  $p$ . Following Arakelyan et al. [2021], Minervini et al. [2022], in our experiments, we use a regularised variant of ComplEx [Trouillon et al., 2016, Lacroix et al., 2018] as the neural link predictor of choice, due to its simplicity, efficiency, and generalisation properties [Ruffinelli et al., 2020]. To ensure that the output of the neural link predictor is always in  $[0,1]$ , following Arakelyan et al. [2021], Minervini et al. [2022], we use either a sigmoid function or min-max re-scaling.
T-norms and Negations Fuzzy logic generalises over Boolean logic by relaxing the logic conjunction  $(\land)$ , disjunction  $(\lor)$  and negation  $(\neg)$  operators through the use of t-norms, t-conorms, and fuzzy negations. A  $t$ -norm  $\top : [0,1] \times [0,1] \mapsto [0,1]$  is a generalisation of conjunction in fuzzy logic [Klement et al., 2000, 2004]. Some examples include the Gödel  $t$ -norm  $\top_{\min}(x,y) = \min\{x,y\}$ , the product  $t$ -norm  $\top_{\mathrm{prod}}(x,y) = x \times y$ , and the Lukasiewicz  $t$ -norm  $\top_{\mathrm{Luk}}(x,y) = \max\{0,x + y - 1\}$ .
Analogously,  $t$ -conorms are dual to  $t$ -norms for disjunctions - given a  $t$ -norm  $\top$ , the complementary  $t$ -conorm is defined by  $\bot (x,y) = 1 - \top (1 - x,1 - y)$ . In our experiments, we use the Gödel  $t$ -norm and product  $t$ -norm with their corresponding  $t$ -conorms.
Fuzzy logic also encompasses negations  $n:[0,1] \mapsto [0,1]$ . The standard  $n_{\mathrm{stand}}(x) = 1 - x$  and strict cosine  $n_{\mathrm{cos}} = \frac{1}{2} (1 + \cos (\pi x))$  are common examples of fuzzy negations[Kruse and Moewes, 1993]. To support a broader class of queries, we introduce the standard and strict cosine functions to model negations in  $\mathbf{CQD}^{\mathcal{A}}$ , which was not considered in the original formulation of CQD.
Continuous Query Decomposition Given a DNF query  $\mathcal{Q}$  as defined in Equation (1), CQD aims to find the variable assignments that render  $\mathcal{Q}$  true. To achieve this, CQD casts the problem of query answering as an optimisation problem. The aim is to find a mapping from variables to entities  $S = \{A\gets a,V_1\gets v_1,\dots ,V_m\gets v_m\}$ , where  $a,v_{1},\ldots ,v_{m}\in \mathcal{E}$  are entities and  $A,V_{1},\ldots ,V_{m}$  are variables, that maximises the score of  $\mathcal{Q}$ :
$$
\underset {S} {\arg \max } \operatorname {s c o r e} (\mathcal {Q}, S) = \underset {A, V _ {1}, \dots , m \in \mathcal {E}} {\arg \max } \left(e _ {1} ^ {1} \top \dots \top e _ {n _ {1}} ^ {1}\right) \perp \dots \perp \left(e _ {1} ^ {d} \top \dots \top e _ {n _ {d}} ^ {d}\right)
$$
$$
\text {w h e r e} e _ {i} ^ {j} = \phi_ {p} \left(\mathbf {e} _ {c}, \mathbf {e} _ {V}\right), \text {w i t h} V \in \{A, V _ {1}, \dots , V _ {m} \}, c \in \mathcal {E}, p \in \mathcal {R} \tag {2}
$$
$$
\text {o r} e _ {i} ^ {j} = \phi_ {p} \left(\mathbf {e} _ {V}, \mathbf {e} _ {V ^ {\prime}}\right), \text {w i t h} V, V ^ {\prime} \in \{A, V _ {1}, \dots , V _ {m} \}, V \neq V ^ {\prime}, p \in \mathcal {R},
$$
where  $\top$  and  $\bot$  denote a t-norm and a t-conorm - a continuous generalisation of the logical conjunction and disjunction, respectively - and  $\phi_p(\mathbf{e}_s,\mathbf{e}_o)\in [0,1]$  denotes the neural link prediction score for the atom  $p(s,o)$ .
Complex Query Answering via Combinatorial Optimisation Following Arakelyan et al. [2021], Minervini et al. [2022], we solve the optimisation problem in Equation (2) by greedily searching for a set of variable substitutions  $S = \{A \gets a, V_1 \gets v_1, \dots, V_m \gets v_m\}$ , with  $a, v_1, \dots, v_m \in \mathcal{E}$ , that maximises the complex query score, in a procedure akin to beam search. We do so by traversing the dependency graph of a query  $\mathcal{Q}$  and, whenever we find an atom in the form  $p(c, V)$ , where  $p \in \mathcal{R}$ ,  $c$  is either an entity or a variable for which we already have a substitution, and  $V$  is a variable for which we do not have a substitution yet, we replace  $V$  with all entities in  $\mathcal{E}$  and retain the top-  $k$  entities  $t \in \mathcal{E}$  that maximise  $\phi_p(\mathbf{e}_c, \mathbf{e}_t)$  - i.e. the most likely entities to appear as a substitution of  $V$  according to the neural link predictor. As we traverse the dependency graph of a query, we keep a beam with the most promising variable-to-entity substitutions identified so far.
Example 3.2 (Combinatorial Optimisation). Consider the query "Which musicians  $M$  received awards associated with a genre  $g$ ?, which can be rewritten as?  $M: \exists A$ . assoc  $(g, A) \land \text{received}(A, M)$ . To answer this query using combinatorial optimisation, we must find the top-  $k$  awards  $a$  that are candidates to substitute the variable  $A$  in assoc  $(g, A)$ . This will allow us to understand the awards associated with the genre  $g$ . Afterwards, for each candidate substitution for  $A$ , we search for the top-  $k$  musicians  $m$  that are most likely to substitute  $M$  in received  $(A, M)$ , ending up with  $k^2$  musicians. Finally, we rank the  $k^2$  candidates using the final query score produced by a t-norm.
4 Calibrating Link Prediction Scores on Complex Queries
The main limitation in the CQD method outlined in Section 3 is that neural link predictors  $\phi$  are trained to answer simple, atomic queries, and the resulting answer scores are not trained to interact with one another.
Example 4.1. Consider the running example query "Which people are German and produced the music for the film Constantine?" which can be rewritten as a complex query  $\mathcal{Q} \equiv ?T$ : country(Germany,  $T$ ) ∧ producerOf(Constantine,  $T$ ). To answer this complex query, CQD answers the atomic sub-queries  $\mathcal{Q}_1 =$  country(Germany,  $T$ ) and  $\mathcal{Q}_2 =$  producerOf(Constantine,  $T$ ) using a neural link predictor, and aggregates the resulting scores using a t-norm. However, the neural link predictor was only trained on answering atomic queries, and the resulting scores are not calibrated to interact with each other. For example, the scores for the atomic queries about the relations country and producerOf may be on different scales, which causes problems when aggregating such scores via t-norms. Let us assume the top candidates for the variable  $T$  coming from the atomic queries  $\mathcal{Q}_1$ ,  $\mathcal{Q}_2$  are  $\mathcal{A}_1 \gets$  Sam Shepard and  $\mathcal{A}_2 \gets$  Klaus Badelt, with their corresponding neural link prediction scores 1.2 and 8.9, produced using  $\phi_{country}$  and  $\phi_{producerOf}$ . We must also factor in the neural link prediction score of the candidate  $\mathcal{A}_1$  for query  $\mathcal{Q}_2$  at 7.4 and vice versa at 0.5. When using the Gödel t-norm  $\top_{\min}(x,y) = \min\{x,y\}$ , the scores associated with the variable assignments  $\mathcal{A}_1$ ,  $\mathcal{A}_2$  are computed as,  $\min(8.0,0.5) = 0.5\min(7.4,1.2) = 1.2$ . For both answers  $\mathcal{A}_1$  and  $\mathcal{A}_2$ , the scores produced by  $\phi_{country}$  for  $\mathcal{Q}_1$  are always lower than the scores produced with  $\phi_{producerOf}$  for  $\mathcal{Q}_2$ , meaning that the scores of the latter are not considered when producing the final answer. This phenomenon can be broadly observed in CQD, illustrated in Figure 2.
To address this problem, we propose a method for adaptively learning to calibrate neural link prediction scores by back-propagating through the complex query-answering process. More formally, let  $\phi_p$  denote a neural link predictor. We learn an additional adaptation function  $\rho_{\theta}$ , parameterised by  $\theta = \{\alpha ,\beta \}$ , with  $\alpha ,\beta \in \mathbb{R}$ . Then, we use the composition of  $\rho_{\theta}$  and  $\phi_p,\rho_\theta \circ \phi_p$ , such that:
$$
\rho_ {\theta} \left(\phi_ {p} \left(\mathbf {e} _ {V}, \mathbf {e} _ {V ^ {\prime}}\right)\right) = \phi_ {p} \left(\mathbf {e} _ {V}, \mathbf {e} _ {V ^ {\prime}}\right) (1 + \alpha) + \beta . \tag {3}
$$
Here, the function  $\rho$  defines an affine transformation of the score and when the parameters  $\alpha = \beta = 0$ , the transformed score  $\rho_{\theta}(\phi_p(\mathbf{e}_V,\mathbf{e}_{V'})))$  recovers the original scoring function. The parameters  $\theta$  can be conditioned on the representation of the predicate  $p$  and the entities  $V$  and  $V'$ , i.e.  $\theta = \psi (\mathbf{e}_V,\mathbf{e}_p,\mathbf{e}_{V'})$ ; here,  $\psi$  is an end-to-end differentiable neural module with parameters  $\mathbf{W}$ .  $\mathbf{e}_V$ ,  $\mathbf{e}_p$ ,  $\mathbf{e}_{V'}$  respectively denote the representations of the subject, predicate, and object of the atomic query. In our experiments, we consider using one or two linear transformation layers with a ReLU non-linearity as options for  $\psi$ .
The motivation for our proposed adaptation function is twofold. Initially, it is monotonic, which is desirable for maintaining the capability to interpret intermediate scores, as in the original formulation of CQD. Moreover, we draw inspiration from the use of affine transformations in methodologies such as Platt scaling [Platt et al., 1999], which also use a linear function for calibrating probabilities and have been applied in the problem of calibration of link prediction models [Tabacof and Costabello, 2020]. Parameter-efficient adaptation functions have also been applied effectively in other domains, such as adapter layers Houlsbry et al. [2019] used for fine-tuning language models in NLP tasks.
Training For training the score calibration component in Equation (3), we first compute how likely each entity  $a^\prime \in \mathcal{E}$  is to be an answer to the query  $\mathcal{Q}$ . To this end, for each candidate answer  $a^\prime \in \mathcal{E}$  we compute the answer score as the complex query score assuming that  $a^\prime \in \mathcal{E}$  is the final answer as:
$$
\operatorname {s c o r e} (\mathcal {Q}, A \leftarrow a ^ {\prime}) = \max  _ {S} \operatorname {s c o r e} (\mathcal {Q}, S), \text {w h e r e} A \leftarrow a ^ {\prime} \in S. \tag {4}
$$
Distribution of neural link prediction scores
Figure 2: The distributions of two atomic scores  $Q_{1}$  and  $Q_{2}$ , and the aggregated results via  $T_{\mathrm{min}} - t$  the scores from  $Q_{2}$  dominate the final scores.