Extending Complex Logical Queries on Uncertain Knowledge Graphs
Authors: Weizhi Fei, Zihao Wang, Hang Yin, Yang Duan, Yangqiu Song
Accepted at: acl25

Extending Complex Logical Queries on Uncertain Knowledge Graphs

Weizhi Fei*,♣, Zihao Wang* †, Hang Yin*, Yang Duan*, Yangqiu Song†
♣ Department of Mathematical Sciences, Tsinghua University, Beijing, China
† CSE, HKUST, HKSAR, China
★ Department of Computer Science, Princeton University, Princeton, United States
{fwz22,h-yin20}@mails.tsinghua.edu.cn
{zwanggc,yqsong}@cse.ust.hk, yd1202@cs.princeton.edu

Abstract
The study of machine learning-based logical query answering enables reasoning with large-scale and incomplete knowledge graphs. This paper advances this area of research by addressing the uncertainty inherent in knowledge. While the uncertain nature of knowledge is widely recognized in the real world, it does not align seamlessly with the first-order logic that underpins existing studies. To bridge this gap, we explore the soft queries on uncertain knowledge, inspired by the framework of soft constraint programming. We propose a neural symbolic approach that incorporates both forward inference and backward calibration to answer soft queries on large-scale, incomplete, and uncertain knowledge graphs. Theoretical discussions demonstrate that our method avoids catastrophic cascading errors in the forward inference while maintaining the same complexity as state-of-the-art symbolic methods for complex logical queries. Empirical results validate the superior performance of our backward calibration compared to extended query embedding methods and neural symbolic approaches.1

1 Introduction
Representing and reasoning with factual knowledge are essential functionalities of artificial intelligence systems. As a powerful way of knowledge representation, Knowledge Graphs (KGs) (Miller, 1995; Suchanek et al., 2007; Vrandečić and Krötzsch, 2014) use nodes to represent entities and edges to encode the relations between entities. Recently, Complex Query Answering (CQA) over KGs has attracted considerable attention because this task requires multi-hop logical reasoning over KGs and supports many applications (Ren et al., 2023; Bai et al., 2024b,a; Hu et al., 2024). This task requires answering the existential First Order Logic (FOL) query, involving existential quantification (\( \exists \)), conjunction (\( \land \)), disjunction (\( \lor \)), and negation (\( \neg \)). While answering FOL queries has been extensively researched by database community (Riesen et al., 2010; Hartig and Heese, 2007), such studies overlook the incompleteness of most KGs. Consequently, conventional graph traversal methods for relational database queries may neglect certain answers due to the missing links of KGs. In recent studies on complex logical queries on knowledge graphs, the generalizability of machine learning models is leveraged to predict the missing links of observed KGs and conduct first-order logic reasoning (Ren and Leskovec, 2020; Arakelyan et al., 2021; Liu et al., 2021; Wang et al., 2023b). This combination of machine learning and logic enables further possibilities in data management (Ren et al., 2023).

The uncertainty of knowledge is widely observed ranging from daily events (Zhang et al., 2020) to complex biological systems (Szkarczyk et al., 2023). To represent the uncertain knowledge, confidence values \( p \) are associated with facts to augment the KG (Carlson et al., 2010; Speer et al., 2017; Szkarczyk et al., 2023), known as the uncertain KG. As exemplified in the right of Figure 1, which illustrates the Job Candidates uncertain KG, confidence values are used to quantify the degree of practice level associated with specific skills. Uncertainty is prevalent in existing KGs primarily because most are constructed using machine learning models, where the predicted likelihood of relational facts inherently introduces uncertainty. Notable examples include O*NET (Pai and Costabello, 2021), STRING (Szkarczyk et al., 2023), ConceptNet (Speer et al., 2017), and AESR (Zhang et al., 2020). To address the incompleteness of uncertain KGs, recent studies estimate the confidence values of missing facts with the generalization power of ML models (Chen et al., 2019; Pai and Costabello, 2021).

1 Equal Contribution
1 Our data and code are available in https://github.com/HKUST-KnowComp/Soft-Queries-on-Uncertain-KG.
Figure 1: (a) Examples of two soft queries in the candidate search procedure. The soft queries introduced in this paper are jointly defined by first-order logic and soft requirements. In particular, soft requirements (necessity and importance) are introduced to characterize fine-grained decision-making preferences, distinguishing them from first-order queries. (b) Incomplete uncertain KG for to what extent a candidate possesses a skill. Solid lines indicate the observed knowledge, while dashed lines indicate the unobserved data. Values indicate confidence level, where the higher value indicates the fact is more likely to be true.

Table 1: Comparison of different problem settings. FO: First Order, EFO: Existential First Order.

<table>
  <tr>
    <th>Problem Settings</th>
    <th>Language of reasoning</th>
    <th>Uncertainty of Knowledge</th>
    <th>Unobserved Knowledge</th>
  </tr>
  <tr>
    <td>Relational database</td>
    <td>FO</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Probabilistic database</td>
    <td>FO</td>
    <td>Confidence value \( p \)</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Open world probabilistic database</td>
    <td>FO</td>
    <td>Confidence value \( p \)</td>
    <td>Uniformed \( p_u \) for all unobserved facts</td>
  </tr>
  <tr>
    <td>Complex logical queries on KG</td>
    <td>EFO</td>
    <td>-</td>
    <td>ML generalization to unobserved facts</td>
  </tr>
  <tr>
    <td>Soft queries on uncertain KG (Ours)</td>
    <td>EFO + Soft requirements</td>
    <td>Confidence value \( p \)</td>
    <td>ML generalization to unobserved facts and confidence \( p \)</td>
  </tr>
</table>

To reason with uncertainty, many extensions of first-order logic have been made to cope with the uncertainty in knowledge representation systems formally (Adams, 1996). It is noteworthy to mention that the study of probabilistic databases also extends the relational databases with a confidence value \( p \in [0, 1] \) (Cavallo and Pittarelli, 1987; Suciu et al., 2022). From a machine learning perspective, however, previous studies in open world probabilistic databases are limited from two aspects: (1) They assume uniform uncertainty for all unobserved knowledge (Ceylan et al., 2021), leading to weaker characterization of the incomplete knowledge without the generalization. (2) They focus on first-order logic, which might be insufficient to describe practical reasoning processes with uncertainty. To address the limitations, we extend the complex query answering to uncertain KG and propose soft queries combining query structure and soft requirements, as shown in the left of Figure 1.

This paper studies the machine learning method for reasoning with incomplete and uncertain knowledge, advancing previous studies in symbolic probabilistic databases. Its contribution is threefold.

Contribution 1: A novel and practical setting. We propose a novel setting of Soft Queries on Uncertain KG (SQUK). Our setting extends the previous setting of complex logical queries on KGs in two ways: (1) For the incomplete knowledge base, SQUK extends the incomplete KG to incomplete and uncertain KG. (2) For the language describing reasoning, SQUK extends first-order language to uncertainty-aware soft queries with soft requirements, which are motivated by real-world reasoning with uncertainty and the establishment in soft constraint programming (Schiet, 1992; Rossi et al., 2006). We also introduce the formal definition in Section 3. The comparison of SQUK against other settings is detailed in Table 1.

Contribution 2: ML method for soft queries. We bridge machine learning and SQUK by proposing Soft Reasoning with calibrated Confidence values (SRC), which uses Uncertain Knowledge Graph Embeddings (UKGEs) to tackle the unobserved information and achieves the same computational complexity as the state-of-the-art inference algorithms (Bai et al., 2023b; Yin et al., 2024). The error analysis is also conducted for SRC given the error bound function, characterizing how the performance is affected by UKGEs and the query structures. Based on our analysis, we suggest calibrating the confidence by debiasing and learning, which further boosts the performance of SRC.

Contribution 3: Extensive empirical studies. We also conduct extensive empirical studies to benchmark the performance of a broad spectrum of methods under the UKGE (Chen et al., 2019) settings. The calibrated SRC is compared against baselines including Query Embedding (Ren and Leskovec, 2020) methods with Number Embeddings (QE+NE) (Vaswani et al., 2017) and symbolic search method (Yin et al., 2024). In particular, we compared the differences between QE+NE and SRC under various soft query settings, demonstrating the advantage of SRC. We also make a fair comparison with large language models on anno-
tated soft queries in a natural language setting.

We highlight the uniqueness of the SQUK setting by examining the differences between uncertain KGs and KGs, comparing soft queries with logical queries, and addressing the challenges posed by two versions of incomplete knowledge. These differences are also summarized in Table 1. Additionally, we present the related work concerning complex logical query answering and uncertain knowledge graph embedding in Appendix A.

2 Background

Uncertain KGs enhance traditional KGs by augmenting each triple fact with a confidence value, thereby facilitating the modeling of uncertain knowledge, which is particularly useful in various domains. Figure 1 illustrates uncertainty in job backgrounds. We formally define an uncertain knowledge graph as a set of knowledge as follows:

Definition 1 (Uncertain knowledge graph). Let \( \mathcal{E} \) be the set of entities and \( \mathcal{R} \) be the set of relations, an uncertain knowledge graph \( \mathcal{G} \) is a set of quadruple \( \{ (s_i, r_i, o_i, p_i) \} \), where \( s_i, o_i \in \mathcal{E} \) are entities, \( r_i \in \mathcal{E} \) is relation and \( p_i \in [0, 1] \) represents the confidence value for the relation fact \( (s_i, r_i, o_i) \). This confidence value \( p_i \) indicates the degree of certainty regarding the truth of the fact.

Following the closed-world assumption (Reiter, 1981) and treating all unobserved facts as false, we can derive the weight graph form for uncertain KG and represent it with the confidence function \( P : \mathcal{E} \times \mathcal{R} \times \mathcal{E} \mapsto [0, 1] \) as follows:

\[
P(h_i, r_i, t_i) = \left\{ \begin{array}{ll}
p_i & (h_i, r_i, t_i, p_i) \in \mathcal{G}, \\
0 & \text{otherwise}.
\end{array} \right.
\]

Uncertain KGs also suffer incomplete issues (Chen et al., 2019, 2021a), with observed knowledge representing only a small portion of the total facts. The assumption that all unseen relational facts are false is inappropriate in real-world scenarios. To address this challenge, previous research on uncertain KGs has proposed a machine learning task to predict the confidence scores of these unseen relational facts (Chen et al., 2019, 2021a). Typically, the observed knowledge in uncertain KGs is split into three nested sets of facts, where \( \mathcal{G}_{train} \subsetneq \mathcal{G}_{valid} \subsetneq \mathcal{G}_{test} \). The training set \( \mathcal{G}_{train} \) is used to train the model, while the validation and test sets are used to evaluate its performance in predicting the confidence scores of unseen facts.

Uncertain Knowledge Graph Embeddings (UKGEs) (Chen et al., 2019, 2021a) have been the mainstream methods for predicting unseen relational facts in uncertain KGs, as they learn low-dimensional representations that effectively capture the semantics between relations and facts, demonstrating strong generalizability. UKGEs are trained on partial facts \( \mathcal{G}_{train} \) and approximate the confidence function \( \mathcal{P} \) deriving from complete facts, defined as the following confidence function:

Definition 2. An UKGE parameterizes a differentiable confidence function \( \hat{P} : \mathcal{E} \times \mathcal{R} \times \mathcal{E} \mapsto [0, 1] \).

In practice, obtaining complete facts is challenging, so \( P_{test} \) induced by \( \mathcal{G}_{test} \) are usually substituted for \( \mathcal{P} \), which adhere to previous approaches (Chen et al., 2019; Pai and Costabello, 2021). We present the connection between this setting and the open-world assumption in Appendix E.

3 Soft Queries

The uncertainty inherent in KGs can be modeled using confidence values for each knowledge. However, current complex logical queries are defined on a boolean basis 3, which is not compatible with uncertain KGs. This uncertainty necessitates new definitions for logical operations and answer sets. In this section, we introduce the definition of our extended soft queries.

3.1 Syntax and semantic

Definition 3 (Syntax of soft queries). Soft queries are the disjunction of soft conjunctive queries \( \phi_i \):

\[
\Phi(y) = \phi_1(y) \otimes \cdots \otimes \phi_q(y),
\]

where \( y \) is the free variable. Each \( \phi_i(y) \) is the conjunction of the soft atomic formula:

\[
\phi_i(y) = \exists x_1, \ldots, x_n. a_{i1} \otimes \cdots \otimes a_{ij}, i = 1, ..., q,
\]

where \( x_1, \ldots, x_n \) represent existentially quantified variables. Each \( a_i \) is a soft atomic formula of the form \( (h, r, t, \alpha, \beta) \) or its negation \( \neg(h, r, t, \alpha, \beta) \). Here, \( r \) denotes the relation, \( h \) and \( t \) can be either an entity in \( \mathcal{E} \) or a variable in \( \{ y, x_1, ..., x_n \} \). \( \alpha \) represents the necessity value, and \( \beta \) represents the importance value. The \( \otimes \) and \( \otimes \) represent soft conjunction and disjunction operation.

3For details on logical queries, please refer to Appendix D.
Definition 4 (Substitution). For a soft query involving variables, the substitution replaces all occurrences of the variable x (or y) with any entity \( s \in \mathcal{E} \) simultaneously, denoted as \( s/x \) (or \( s/y \)).

We denote \( \phi(s) \) for the result of substituting s for the free variable y. When all variables in the soft query \( \phi \) have been substituted, we refer to it as the substituted query. Next, we define the semantics of the soft queries, starting with the soft atomic formula. Specially, the soft atomic formula involves two novelty concepts: \( \alpha \) necessity and \( \beta \) importance, which are inspired form soft Constraint Satisfaction Problems (CSPs) (Rossi et al., 2006) to manipulate the uncertainty of facts. We introduce the related work of soft CSPs in Appendix F.

1. The \( \alpha \) necessity component draws inspiration from possibilistic CSPs (Schiex, 1992) and is designed to capture necessity criteria. It serves the purpose of filtering out unnecessary constraints and involves a thresholding operation. The thresholding operation \( [p]_{\alpha} \) is defined as follows:

\[
[p]_{\alpha} = \left\{ \begin{array}{ll}
p & p \geq \alpha, \\
0 & \text{otherwise}.
\end{array} \right.
\]

2. The \( \beta \) importance component is influenced by weighted CSPs (Bistarelli et al., 1999) to describe preference, which is the weight employed to adjust the relative significance of different conditions.

Definition 5 (Semantic of soft queries). Given a semiring \( (\mathbb{R}^+, \oplus, \otimes, \mathbf{0}) \) over \( \mathbb{R}^+ \), the confidence function \( P \) induced by an uncertain knowledge graph \( \mathcal{G} \), and a soft query \( \phi \), let s and o be entities in \( \mathcal{E} \). The confidence value \( U(\phi, P) \) is recursively defined as follows:

1. If \( \phi \) is the substituted soft atomic query \( (s, r, o, \alpha, \beta) \), then \( U(\phi, P) = \beta [P(s, r, o)]_{\alpha} \).
2. If \( \phi \) is the negation of the substituted soft atomic \( \neg(s, r, o, \alpha, \beta) \), then \( U(\phi, P) = \beta [1 - P(s, r, o)]_{\alpha} \);
3. If \( \phi = \exists x_i \psi(y; x_i) \) is the soft query involving existentially quantified variables, then \( U(\phi, P) = \oplus_{s \in \mathcal{E}} U(\phi(y; s/x_i), P) \);
4. If \( \phi \) is the conjunctive query \( (\phi_1 \otimes \phi_2) \), then \( U(\phi, P) = U(\phi_1, P) \otimes U(\phi_2, P) \).
5. If \( \Phi \) is the disjunctive query \( (\Phi_1 \vee \Phi_2) \), then \( U(\Phi, P) = U(\Phi_1, P) \oplus U(\Phi_2, P) \).

To align with the semantics of the confidence value, we instantiate the semiring as \( (\otimes, \oplus, \mathbf{0}) = (+, \max, -\infty) \). We discuss the utilized semiring of the previous soft CSP setting in Appendix F. With semantics, we can compute the utility of entity s for the soft query \( \phi \). The utility of all entities can be conveniently represented as a vector.

Definition 6 (Utility vector). Given a confidence function \( P \) induced by an uncertain knowledge base and a soft query \( \phi \), the utility vector of soft query \( \Phi \), denoted as \( \mathbf{u} \in \mathbb{R}^{|\mathcal{E}|} \), is defined as:

\[
\mathbf{u}_i = U(\Phi(s_i/y), P),
\]

where \( s_i \) denotes the entity indexed by i.

3.2 Example to explain the soft queries, as well as the necessity and importance

Soft queries are a powerful tool for modeling candidate searches across various job positions, enabling nuanced assessments of qualifications. To illustrate this, we present an example of using soft queries to model candidate searches for two roles: Junior Software Developer (JSD) and Principal Investigator (PI) in machine learning. Let HAS denote the relation describing a candidate’s possession of a skill, while LEAD, DEV, and ML represent leadership, development, and machine learning skills, respectively. Regarding leadership, the Principal Investigator places a significantly high emphasis on it, while the Junior Software Developer does not. This distinction is captured by the importance parameter \( \beta \), which assigns greater weight to leadership in the query for the Principal Investigator. Additionally, the necessity parameter \( \alpha \) serves as a threshold to filter out candidates who lack the required skills. The two roles can be modeled using the following soft queries, as further explained in Figure 1:

\[
\phi_{\text{JSD}}(y) = -(y, \text{HAS}, \text{LEAD}, 0.7, 1)) \otimes (y, \text{HAS}, \text{DEV}, 0.5, 3),
\]
\[
\phi_{\text{PI}}(y) = (y, \text{HAS}, \text{LEAD}, 0.7, 3) \otimes (y, \text{HAS}, \text{ML}, 0.9, 1)).
\]

Beyond recruitment, soft queries provide an effective and tailored strategy for modeling logical queries on uncertain KGs. This capability can extend the application of logical queries to uncertain KGs, enabling use cases such as product recommendation (Bai et al., 2024b) and understanding user intentions (Bai et al., 2024a).

3.3 Soft query graph and utility vector

Definition 7 (Soft query graph). Given a soft query \( \phi(y; x_1, ..., x_n) = \exists x_1, ..., x_n a_1 \otimes \cdots \otimes a_m \), the soft query graph \( G_{\phi} \) is defined by tuples induced by soft atomic formulas or its negation: \( G_{\phi} = \{(h_i, r_i, t_i, \alpha_i, \beta_i, \text{NEG}_i)\}_{i=1}^m \),
where \( (h_i, r_i, t_i, \alpha_i, \beta_i, \text{NEG}_i) \) is induced by \( (h_i, r_i, t_i, \alpha_i, \beta_i) \) or \( -(h_i, r_i, t_i, \alpha_i, \beta_i) \). \( \text{NEG}_i \) is the bool variable indicating if \( a_i \) is negated.

If \( h \) (or \( t \)) is a variable, we say the corresponding node in \( G_\phi \) is a variable node. \( V(G_\phi) \) indicates the set of all variable nodes in \( G_\phi \). If \( h \) (or \( t \)) is an entity, we say the corresponding node is an constant node. The leaf node is a node that is only connected to one other node in the query graph. Compared to the operation tree in complex logical queries (Ren and Leskovec, 2020), the soft query graph can model any conjunctive soft queries.

4 Methodology

In this section, we propose Soft Reasoning with calibrated Confidence values (SRC) to facilitate reasoning with various query structures and soft requirements. SRC is a symbolic reasoning method that utilizes UKGE to provide confidence values. Since UKGE inevitably has prediction errors, we present a mild assumption regarding the UKGE error bound in Equation (7). Our error analysis over SRC indicates that the inference error is manageable as the complexity of the query structure increases. To further reduce this error, we introduce two orthogonal calibration strategies: Debiasing (D) and Learning (L).

4.1 Forward inference

The main paper discusses soft queries in which the query graphs are acyclic simple graphs. Cases with the complete case (cycles and self-loops) are detailed in Appendix H.

Given the soft query \( \phi \), SRC efficiently derives the utility vector \( U(\phi, \hat{P}) \) based on the confidence function \( \hat{P} \) approximated by UKGE. The core idea is to progressively prune the edges of the soft query graph while preserving the constraints of the remaining edges, ensuring that the final utility vector remains unchanged. State vectors are used to record the constraints of the pruned edges during the inference process. Specifically, each variable node \( z \in G_\phi \) is described by a state vector \( C_z \in \mathbb{R}^{|\mathcal{E}|} \). The notation \( (G_\phi, \{C_z : z \in V(G_\phi)\}) \) denotes a soft query graph with state vectors.

We define equivalent transformations as \( T \):

\[
T(G_\phi, \{C_z : z \in V(G_\phi)\}) = (G_\psi, \{C'_z : z \in V(G_\psi)\}),
\]

where \( G_\psi \) is a subgraph of \( G_\phi \) (with at least one edge eliminated), \( C'_z \) is the updated state vector, and \( T \) guarantees the utility vector \( \hat{u} \) unchanged.

Algorithm 1 SRC (simple acyclic case)

Require: Input soft query graph \( G_\phi \) and initialize the state vectors \( \{C_z\} \).
Ensure: Output utility vector \( \hat{u}(G_\phi, \{C_z\}) \).
\(
(G_\phi, \{C_z\}) \leftarrow \text{REMOVECONSTNODE}(G_\phi, \{C_z\})
\)
while There exists a leaf node do
\(
(G_\phi, \{C_z\}) \leftarrow \text{REMOVELEAFNODE}(G_\phi, \{C_z\})
\)
end while
Get the utility vector by retrieving \( C_y \).

Two lemmas are presented to induce two equivalent transformations, denoted as \( T_e \) and \( T_l \), respectively. The proof can refer to Appendix H.

Lemma 1. For each constant node in \( G_\phi \), an \( O(|\mathcal{E}|) \) transformation \( T_e \) exists to remove it.

Realization of the equivalent transformation \( T_e \) induces a function REMOVECONSTNODE.

Lemma 2. For each leaf node in \( G_\phi \), an \( O(|\mathcal{E}|^2) \) transformation \( T_l \) exists to remove it.

Realization of the equivalent transformation \( T_l \) induces a function REMOVELEAFNODE.

The above two transformations are constructive and can be applied to remove edges once the corresponding nodes are found. Until the soft query graph only contains a free variable node, the state vector of the free variable \( C_y \) is the desired utility vector. The procedure of SRC for acyclic soft query graphs is presented in Algorithm 1 and one toy example of the execution is visualized in Figure 3. We first remove constant nodes, as these are commonly found and can be easily eliminated using Lemma 1. Subsequently, we identify and remove leaf nodes step by step, according to the Lemma 2, noting that the leaf node always exists for acyclic queries.

Complexity analysis. We begin with a rough estimation of the complexity. The space complexity of the inference algorithm is \( O(|\mathcal{R}||\mathcal{E}|^2) \). Let \( n_e \) denote the number of edges involving existential variables and \( n_r \) represent the remaining edges. For acyclic queries, the time complexity is \( O(n_e|\mathcal{E}|^2 + n_r|\mathcal{E}|) \). We further reduce complexity by leveraging sparsity, a natural characteristic of knowledge graphs. For space complexity, only non-zero values or those exceeding a threshold \( \delta_1 \) are stored and the average sparsity ratio of our constructed neural matrices is approximately 3.4%, reducing storage requirements by 97%. For time complexity, sparsity can be used to accelerate computation. Removing a leaf node \( x \) typi-
cally involves \( O(E^2) \) operations, but by considering only rows where \( C_x \) is non-zero, this reduces to \( O(\mathcal{E} \cdot (|C_x > \delta_2|)) \), where \( |C_x > \delta_2| \) represents the non-zero number of \( C_x \). Thus, the time complexity becomes \( O(n_e \mathcal{E} \cdot \max_x(|C_x > \delta_2|)) \), where \( n_e \) is the number of edges involving existential variables.

4.2 Error analysis

The current model still exhibits significant prediction errors, with the mean absolute error on CN15K generally reaching around 0.2 (Chen et al., 2021a,b). To facilitate the error analysis of our proposed search algorithm, we introduce the error bound \( \varepsilon(\delta) \) as the following:

Definition 8. Let \( \varepsilon \) be a function that maps the error \( \delta \) to a tail probability which can uniformly bounds the error of \( \hat{P} \) for some kind of norm:

\[
\Pr \left( \max_{(s,r,o)} \| \hat{P}(s,r,o) - \mathcal{P}(s,r,o) \| > \delta \right) < \varepsilon(\delta).
\]

We note that \( \varepsilon(\delta) < 1 \) is guaranteed for all \( \delta \) and \( \varepsilon(\delta) \) decreases monotonically with \( \delta \), though a better link predictor provides a tighter \( \varepsilon(\delta) \). This definition provides a practical and flexible approach for analyzing the errors over uncertain KGs. We look into the error of each soft atomic query:

Theorem 1. For any soft atomic query \( \psi = (h, r, y, \alpha, \beta) \), let the uniform inference error be

\[
\max_{\psi,s \in \mathcal{E}} \| U(\psi(s/y), \hat{P}) - U(\psi(s/y), \mathcal{P}) \| = \epsilon(\alpha, \beta)
\]

Then we estimate the distribution of \( \epsilon(\alpha, \beta) \) by the uniform error-bound \( \varepsilon(\delta) \) provided in Equation (7) and assume the probability density function of \( \mathcal{P} \) is \( f(\xi) \):

\[
\Pr (\epsilon(\alpha, \beta) > \delta) < \varepsilon(\frac{\delta}{\beta}) + (1 - \varepsilon(\frac{\delta}{\beta})) \int_{\delta}^{1} \varepsilon(|\alpha - \xi|) f(\xi) d\xi.
\]

Moreover, the numerical stability is guaranteed:

Theorem 2. For a soft conjunctive query \( \phi = \exists x_1, ..., x_n, a_1 \otimes \cdots \otimes a_m \), where \( a_i = (h_i, r_i, t_i, \alpha_i, \beta_i) \), and any entity \( s \in \mathcal{E} \), the error accumulated is at most linear:

\[
\| U(\phi(s), \hat{P}) - U(\phi(s), \mathcal{P}) \| \leq \sum_{i=1}^{m} \epsilon(\alpha_i, \beta_i).
\]

This conclusion ensures that there is no catastrophic cascading error in our forward inference algorithm. The proof of all the above theorems can refer to Appendix N.

4.3 Two calibration strategies

Debiasing. The confidence function \( \hat{P} \) of UKGE is biased towards zero. We propose a debiasing strategy for the inference. We modify the soft requirements \( \alpha \) as \( \alpha - \Delta_\alpha \). We can see that this simple debiasing strategy improves performance. SRC with this strategy is denoted as SRC(D).

Learning. The pre-trained UKGE is not optimal for SRC in the incomplete uncertain KGs. We propose the calibration by learning (L) strategy by learning the calibrated confidence function. Specifically, we calibrate the confidence function \( P_c \) by learnable affine transformation (Arakelyan et al., 2023) as following:

\[
\hat{P}_c(s, r, o) = \hat{P}(s, r, o)(1 + \rho_\theta(s, r, o)) + \lambda_\theta(s, r, o),
\]
\[
[\rho_\theta(s, r, o), \lambda_\theta(s, r, o)] = \sum_{j \in \{s, r, o\}} (W_j e_j + b_j).
\]

Here, \( \rho_\theta(s, r, o) \) and \( \lambda_\theta(s, r, o) \) are the affine parameters. \( e_j \in \mathbb{R}^d \) represents the embedding for entity or relation \( j \in \{s, r, o\} \) with embedding dimension \( d \). Meanwhile, \( W_j \in \mathbb{R}^{2 \times d} \) and \( b_j \in \mathbb{R}^2 \) are the learnable parameters associated with \( j \).

As implied by both Theorem 1 and Theorem 2, the error of SRC is rooted in the error bound of UKGE. As we can see from Equation (8) the error bound is governed by both \( \varepsilon \) and the integral of \( \varepsilon \) over the domain \( [0, \max(\alpha, 1 - \alpha)] \). An important implication is that when \( \alpha = 0 \), the integral of \( \varepsilon \) will be fully \( [0, 1]^4 \). Therefore, our theoretical analysis motivates the goal of calibration as the minimization of the mean squared error between

![A toy model illustrating the process of the SRC. Each variable node is assigned a state vector, which is updated through the algorithm as edges are removed. The final state vector of the free variable is the desired.](page_120_120_600_240.png)

Figure 2: A toy model illustrating the process of the SRC. Each variable node is assigned a state vector, which is updated through the algorithm as edges are removed. The final state vector of the free variable is the desired.
the predicted utility and the observed utility of answers:

\[
\mathcal{L} = \sum_{u(s)>0} (U(\phi(s), \hat{P}_c) - U(\phi(s), \mathcal{P}))^2,
\]

where \( U(\phi(s), \hat{P}_c) \) represents the predicted utility vector of a soft query \( \phi \) according to Definition 6. SRC with this strategy is denoted as SRC(L).

Notably, we only need to train the calibration transformation in the cases of \( \alpha = 0 \), achieving a simpler training strategy but better generalization capability when compared to the QE+NE baselines, as will be presented in Appendix B.

5 SQUK Dataset Construction

We provide a brief overview of dataset construction and the details can refer to Appendix K.

5.1 Useful queries and evaluation protocols

The validation/test uncertain knowledge graph incorporates new facts that will update the utility vectors of specific soft queries. Only these particular queries are considered meaningful and included in the evaluation. The evaluation of soft queries not only considers recall but also accounts for the values of the recalled answers. Therefore, we adopt metrics from the learning-to-rank framework (Liu et al., 2009) as our evaluation protocol, which includes Mean Average Precision (MAP), Normalized Discounted Cumulative Gain (NDCG), Spearman’s rank correlation coefficient (\( \rho \)), and Kendall’s rank correlation coefficient (\( \tau \)).

We choose 1P, 2P, 2I, 2IN, and 2IL as train query types and add 3IN, INP, IP, 2M, and IM as validation/test query types. The training query types encompass basic operations, allowing us to evaluate the ability of machine learning methods to generalize to commonly used unseen query types (Yin et al., 2024). We visualize the structure of these types in Figure 4. Additionally, Table 8 shows the statistics of training queries of our dataset.

Table 2: The statistics of train queries.

<table>
  <tr>
    <th>KG</th>
    <th>1P</th>
    <th>2P</th>
    <th>2I</th>
    <th>2IN</th>
    <th>2IL</th>
  </tr>
  <tr>
    <td>PPI5k</td>
    <td>9,724</td>
    <td>9,750</td>
    <td>9,754</td>
    <td>1,500</td>
    <td>1,500</td>
  </tr>
  <tr>
    <td>O*NET20k</td>
    <td>18,266</td>
    <td>18,300</td>
    <td>18,300</td>
    <td>1,850</td>
    <td>1,850</td>
  </tr>
  <tr>
    <td>CN15k</td>
    <td>52,887</td>
    <td>52,900</td>
    <td>52,900</td>
    <td>5,300</td>
    <td>5,300</td>
  </tr>
</table>

5.2 Uncertain KGs

We utilize three standard uncertain KGs: CN15k (Chen et al., 2019) for encompassing commonsense, PPI5k (Chen et al., 2019) for biology, and O*NET20K (Pai and Costabello, 2021) for employment domains. These uncertain KGs are noisy and incomplete, requiring the ML models to predict the confidence values.

5.3 Soft requirements

For the \( \alpha \) parameter, we establish connections with the percentile value of the relation to represent the necessity value effectively. We assign specific percentiles to different necessity levels: the 25th, 50th, and 75th percentiles correspond to “low”, “normal”, and “high” necessity criteria, respectively. We ensure that a “zero” requirement is assigned when the necessity criteria reaches 0. We also introduce a hybrid strategy that randomly selects necessity values, enabling a comprehensive evaluation.

For the \( \beta \) importance setting, we employ two strategies: “equal” and “random”. Under the “equal” strategy, all importance values are assigned an importance value of 1.0. In contrast, the “random” strategy introduces variability by assigning random decimal numbers between 0 and 1 to represent the importance of each soft atomic formula.

Soft queries feature flexible settings for \( \alpha \) and \( \beta \), allowing users to customize them according to their varying needs. To facilitate the evaluation of our method’s practicality in real-world situations, the proposed “random” strategy simulates a diverse range of query options.

6 Experiments

In this section, we empirically explore how to answer soft queries. We mainly compare our method with generalized SoTA CQA models on SQUK dataset, including commonly used query embedding models and advanced symbolic search methods. Additionally, we evaluate the performance of advanced commercial LLMs on soft queries with clear natural language descriptions. The implementation details of these experiments are in Appendix G. We also conduct the ablation study regarding the distribution and impact of two parameters \( \alpha \) and \( \beta \) on both kinds of approaches, which is presented in Appendix B due to page limitations. We also provide the qualitative analysis of the experiments in Appendix C.

6.1 Main results

Baselines We select two mainstream CQA methods as baselines: query embedding and symbolic
Table 3: Result of answering soft queries. Logic+NE and ConE+NE refer to the query embedding with number embedding extensions. SRC is our inference method, and SRC(D), SRC(L), and SRC(D+L) are explained in Section 4.3. The four metrics are all higher, indicating better performance.

<table>
  <tr>
    <th rowspan="2">Uncertain KG</th>
    <th rowspan="2">Models</th>
    <th colspan="12">τ</th>
    <th colspan="3">AVG.</th>
  </tr>
  <tr>
    <th>IP</th><th>2P</th><th>2I</th><th>2IN</th><th>2IL</th><th>2M</th><th>2U</th><th>3IN</th><th>IP</th><th>IM</th><th>INP.</th><th>UP.</th><th>AVG.</th><th>AVG. p</th><th>AVG. MAP</th><th>AVG. NDCG</th>
  </tr>
  <tr>
    <td rowspan="6">CN15k</td>
    <td>Logic+NE</td><td>9.1</td><td>-1.5</td><td>4.8</td><td>6.0</td><td>18.3</td><td>5.1</td><td>14.1</td><td>3.5</td><td>2.4</td><td>6.4</td><td>-0.9</td><td>9.6</td><td>4.8</td><td>5.8</td><td>7.0</td><td>11.2</td>
  </tr>
  <tr>
    <td>ConE+NE</td><td>5.3</td><td>4.3</td><td>3.5</td><td>6.3</td><td>18.4</td><td>6.9</td><td>20.5</td><td>2.9</td><td>1.8</td><td>10.4</td><td>1.7</td><td>14.9</td><td>8.1</td><td>10.0</td><td>7.7</td><td>13.2</td>
  </tr>
  <tr>
    <td>LMPNN</td><td>22.8</td><td>9.2</td><td>4.8</td><td>10.9</td><td>15.5</td><td>9.7</td><td>9.7</td><td>4.3</td><td>5.7</td><td>9.1</td><td>4.9</td><td>1.5</td><td>9.0</td><td>11.4</td><td>14.7</td><td>21.2</td>
  </tr>
  <tr>
    <td>SRC</td><td>15.0</td><td>2.1</td><td>-0.0</td><td>2.1</td><td>1.7</td><td>9.2</td><td>25.5</td><td>2.0</td><td>-0.9</td><td>7.9</td><td>-4.4</td><td>13.0</td><td>5.9</td><td>8.9</td><td>9.2</td><td>15.5</td>
  </tr>
  <tr>
    <td>SRCD</td><td>16.6</td><td>11.8</td><td>-0.4</td><td>10.6</td><td>10.9</td><td>11.7</td><td>14.1</td><td>4.0</td><td>3.2</td><td>13.5</td><td>1.7</td><td>11.5</td><td>13.1</td><td>12.9</td><td>21.8</td>
  </tr>
  <tr>
    <td>SRCL</td><td>15.8</td><td>11.8</td><td>-0.4</td><td>2.4</td><td>11.0</td><td>12.4</td><td>32.3</td><td>-0.8</td><td>3.6</td><td>11.1</td><td>1.1</td><td>22.8</td><td>10.3</td><td>13.7</td><td>12.6</td><td>21.1</td>
  </tr>
  <tr>
    <td rowspan="6">PP15k</td>
    <td>Logic+NE</td><td>20.5</td><td>22.6</td><td>17.1</td><td>10.4</td><td>24.4</td><td>20.0</td><td>30.4</td><td>9.1</td><td>12.4</td><td>14.1</td><td>-2.6</td><td>32.9</td><td>14.8</td><td>20.7</td><td>8.0</td><td>16.4</td>
  </tr>
  <tr>
    <td>ConE+NE</td><td>29.2</td><td>42.5</td><td>26.4</td><td>20.7</td><td>32.6</td><td>33.9</td><td>35.9</td><td>16.6</td><td>36.6</td><td>29.4</td><td>22.5</td><td>42.2</td><td>30.7</td><td>40.8</td><td>44.1</td><td>49.2</td>
  </tr>
  <tr>
    <td>LMPNN</td><td>30.1</td><td>38.1</td><td>20.2</td><td>14.0</td><td>34.0</td><td>27.2</td><td>18.0</td><td>13.7</td><td>34.3</td><td>27.5</td><td>25.7</td><td>29.1</td><td>26.0</td><td>35.2</td><td>32.0</td><td>39.0</td>
  </tr>
  <tr>
    <td>SRC</td><td>46.6</td><td>38.9</td><td>29.9</td><td>27.2</td><td>47.1</td><td>47.3</td><td>37.7</td><td>37.7</td><td>47.1</td><td>47.2</td><td>47.4</td><td>48.2</td><td>48.2</td><td>68.4</td><td>70.7</td><td>70.7</td>
  </tr>
  <tr>
    <td>SRCD</td><td>66.7</td><td>68.7</td><td>53.1</td><td>44.9</td><td>73.1</td><td>46.7</td><td>73.3</td><td>37.7</td><td>63.7</td><td>41.3</td><td>56.9</td><td>69.9</td><td>58.0</td><td>68.5</td><td>64.0</td><td>69.8</td>
  </tr>
  <tr>
    <td>SRCL</td><td>66.8</td><td>71.7</td><td>52.7</td><td>43.4</td><td>72.8</td><td>43.8</td><td>72.7</td><td>34.4</td><td>66.6</td><td>38.3</td><td>58.0</td><td>71.4</td><td>57.7</td><td>67.8</td><td>69.8</td><td>71.6</td>
  </tr>
  <tr>
    <td rowspan="6">O*NET20k</td>
    <td>Logic+NE</td><td>6.3</td><td>9.5</td><td>43.5</td><td>3.9</td><td>36.6</td><td>9.7</td><td>15.3</td><td>8.3</td><td>11.1</td><td>8.8</td><td>3.8</td><td>-9.8</td><td>13.8</td><td>18.5</td><td>3.5</td><td>6.4</td>
  </tr>
  <tr>
    <td>ConE+NE</td><td>30.8</td><td>41.7</td><td>57.0</td><td>41.4</td><td>46.6</td><td>37.9</td><td>40.7</td><td>48.0</td><td>22.1</td><td>21.7</td><td>14.2</td><td>53.1</td><td>38.4</td><td>47.2</td><td>27.5</td><td>48.7</td>
  </tr>
  <tr>
    <td>LMPNN</td><td>26.0</td><td>39.9</td><td>12.9</td><td>12.7</td><td>17.5</td><td>18.9</td><td>29.2</td><td>17.0</td><td>29.5</td><td>17.1</td><td>23.9</td><td>13.9</td><td>13.8</td><td>21.2</td><td>21.2</td>
  </tr>
  <tr>
    <td>SRC</td><td>72.0</td><td>54.9</td><td>68.6</td><td>67.6</td><td>67.3</td><td>36.9</td><td>76.0</td><td>59.2</td><td>47.6</td><td>29.1</td><td>48.9</td><td>52.4</td><td>57.3</td><td>65.3</td><td>27.1</td><td>41.3</td>
  </tr>
  <tr>
    <td>SRCD</td><td>71.7</td><td>55.2</td><td>74.3</td><td>67.7</td><td>70.3</td><td>49.3</td><td>80.1</td><td>65.0</td><td>52.7</td><td>44.7</td><td>48.9</td><td>55.2</td><td>61.8</td><td>70.2</td><td>26.6</td><td>41.5</td>
  </tr>
  <tr>
    <td>SRCL</td><td>71.6</td><td>55.6</td><td>64.3</td><td>67.5</td><td>71.0</td><td>49.7</td><td>80.2</td><td>65.0</td><td>52.9</td><td>45.2</td><td>49.2</td><td>55.9</td><td>61.9</td><td>70.5</td><td>26.7</td><td>41.8</td>
  </tr>
  <tr>
    <td>SRCD+L</td><td>71.7</td><td>55.6</td><td>64.3</td><td>67.5</td><td>71.0</td><td>49.7</td><td>80.2</td><td>65.0</td><td>52.9</td><td>45.2</td><td>49.2</td><td>55.9</td><td>61.9</td><td>70.5</td><td>26.7</td><td>41.7</td>
  </tr>
</table>

search. Specifically, we focus on two classical query embedding methods: LogicE (Luus et al., 2021) and ConE (Zhang et al., 2021). To enable soft queries, we incorporate the relation projection network with Number Embedding (NE) and adjust the loss function accordingly.5 The forward inference of our method, SRC, is directly generalized from SoTA symbolic methods FIT (Yin et al., 2024), which serve as baselines for search methods.

Models analysis. The main results are presented in Table 3, demonstrating that our proposed method significantly outperforms both query embedding methods and directly generalized symbolic methods. By leveraging the two calibration strategies, debiasing (D) and learning (L), as explained in Section 4.3, our method achieves superior results across most KGs and metrics on average.

Query structure analysis. Although ConE performs well on some trained query types, it struggles with newly emerged query types and those involving negation, such as INP and IM. In contrast, our method exhibits excellent performance across the majority of query types, demonstrating robust combinatorial generalization capabilities on complex queries. Our method particularly excels in handling challenging query types that involve existential variables, such as 2P, 2M, IM, and INP, highlighting its advantages in these scenarios.

6.2 The comparison with LLMs

We devise an evaluation framework to assess the performance of LLMs, benchmarking their powerful reasoning abilities over uncertain knowledge. To ensure fairness of the comparison, we consider the queries sampled from CN15k and we choose four candidate answers for each query. These queries have also been manually filtered and labeled to ensure clearness and correctness. We describe the syntax and semantics of soft queries using natural language, prompting LLMs to select the most suitable answer. The details on this setting construction can refer to Appendix M.

The results, shown in Table 4, indicate that even the simple symbolic SRC achieves significantly higher accuracy compared to Llama3.3 70B, Gemini-1.5-pro, GPT-3.5-turbo, and GPT-4-preview. This demonstrates that large language models (LLMs) struggle with complex arithmetic operations involving uncertain values of knowledge. Our evaluation is fair, as the required uncertain knowledge is derived from well-known commonsense KGs ConceptNet, and the logical operations are expressed in natural language. Nevertheless, even advanced commercial LLMs struggle to select the highest-scoring answer. This further emphasizes the difficulties presented by the proposed soft queries and highlights the ongoing need for the development of symbolic approaches.

Table 4: The accuracy of manually annotated queries.

<table>
  <tr>
    <th>Model</th>
    <th>Llama3.3 70B</th>
    <th>Gemini-1.5-pro</th>
    <th>GPT-3.5-turbo</th>
    <th>GPT-4-preview</th>
    <th>SRC</th>
  </tr>
  <tr>
    <th>Accuracy</th>
    <td>42.6</td>
    <td>37.1</td>
    <td>34.3</td>
    <td>37.8</td>
    <td>48.9</td>
  </tr>
</table>

5Detailed information can be found in Appendix J.
7 Conclusion

In this paper, we introduce a novel setting, soft queries on uncertain KGs, which further extends complex logical queries on KG. The soft queries consider the incompleteness of large-scale uncertain KGs and require the incorporation of ML methods to estimate scores for new relational linking while handling semiring algebraic structures. Our proposed soft queries also propose the soft requirements inspired by soft constraint satisfaction problems to control the uncertainty of knowledge. To facilitate the research of soft queries, we construct a soft query answering dataset consisting of three uncertain KGs. Furthermore, we propose a new neural-symbolic approach with both forward inference and backward calibration. Both theoretical analysis and experimental results demonstrate that our method has satisfactory performance.

8 Limitation

Soft queries extend complex logical queries over Knowledge Graphs (KGs) by incorporating soft requirements within uncertain KGs. However, the scope of the proposed soft queries is limited, as it primarily focuses on conjunctive queries. While conjunctive queries form the foundation of complex logical queries, this restriction may hinder the expressiveness and applicability of the proposed soft queries. Furthermore, the dataset does not include cyclic queries, which are NP-complete, even though their complexity can be more easily addressed.

9 Potential Impact

Soft queries have the potential to perpetuate existing biases present in the underlying knowledge graphs. If these graphs contain skewed or discriminatory information, the results generated by soft queries may reflect and amplify these biases, leading to unfair outcomes in applications such as hiring, credit scoring, or law enforcement. This raises significant ethical concerns about fairness, as marginalized groups may be disproportionately affected by biased query results, resulting in systemic inequality.

The utilization of soft queries to extract information from knowledge graphs can pose serious privacy risks. If queries access sensitive personal data without proper safeguards, there is a potential for unauthorized disclosures that violate individuals’ privacy rights. This concern is heightened in contexts where the data might be used for profiling or surveillance, making it imperative to establish robust privacy protections and ethical guidelines to ensure that individuals’ information is handled responsibly and transparently.

10 Acknowledge

The authors of this paper were supported by the ITSP Platform Research Project (ITS/189/23FP) from ITC of Hong Kong, SAR, China, and the AoE (Aoe/E-601/24-N), the RIF (R6021-20) and the GRF (16205322) from RGC of Hong Kong,SAR, China.

References

Ernest Wilcox Adams. 1996. A primer of probability logic.

Alfonso Amayuelas, Shuai Zhang, Xi Susie Rao, and Ce Zhang. 2021. Neural Methods for Logical Reasoning over Knowledge Graphs.

Erik Arakelyan, Daniel Daza, Pasquale Minervini, and Michael Cochez. 2021. Complex Query Answering with Neural Link Predictors. In International Conference on Learning Representations.

Erik Arakelyan, Pasquale Minervini, and Isabelle Augenstein. 2023. Adapting Neural Link Predictors for Complex Query Answering. arXiv preprint. ArXiv:2301.12313 [cs].

Jiaxin Bai, Xin Liu, Weiqi Wang, Chen Luo, and Yangqiu Song. 2023a. Complex query answering on eventuality knowledge graph with implicit logical constraints. Advances in Neural Information Processing Systems, 36:30534–30553.

Jiaxin Bai, Chen Luo, Zheng Li, Qingyu Yin, and Yangqiu Song. 2024a. Understanding inter-session intentions via complex logical reasoning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 71–82.

Jiaxin Bai, Yicheng Wang, Tianshi Zheng, Yue Guo, Xin Liu, and Yangqiu Song. 2024b. Advancing abductive reasoning in knowledge graphs through complex logical hypothesis generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1312–1329.

Jiaxin Bai, Zihao Wang, Hongming Zhang, and Yangqiu Song. 2022. Query2Particles: Knowledge Graph Reasoning with Particle Embeddings. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2703–2714.
Jiaxin Bai, Zihao Wang, Yukun Zhou, Hang Yin, Weizhi Fei, Qi Hu, Zheye Deng, Jiayang Cheng, Tianshi Zheng, Hong Ting Tsang, et al. 2025. Top ten challenges towards agentic neural graph databases. arXiv preprint arXiv:2501.14224.

Yushi Bai, Xin Lv, Juanzi Li, and Lei Hou. 2023b. Answering Complex Logical Queries on Knowledge Graphs via Query Computation Tree Optimization. In Proceedings of the 40th International Conference on Machine Learning, pages 1472–1491. PMLR. ISSN: 2640-3498.

Stefano Bistarelli, Ugo Montanari, Francesca Rossi, Thomas Schietex, Gérard Verfaillie, and Hélène Fargier. 1999. Semiring-based csp and valued csp: Frameworks, properties, and comparison. Constraints, 4:199–240.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multirelational Data. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.

Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam Rruschka, and Tom Mitchell. 2010. Toward an architecture for never-ending language learning. In Proceedings of the AAAI conference on artificial intelligence, volume 24, pages 1306–1313.

Roger Cavallo and Michael Pittarelli. 1987. The theory of probabilistic databases. In VLDB, volume 87, pages 1–4.

Ismail Ilkan Ceylan, Adnan Darwiche, and Guy Van den Broeck. 2021. Open-world probabilistic databases: Semantics, algorithms, complexity. Artificial Intelligence, 295:103474.

Jianshu Chen. 2023. Learning Language Representations with Logical Inductive Bias. arXiv preprint ArXiv:2302.09458 [cs].

Xuelu Chen, Michael Boratko, Muhao Chen, Shib Sankar Dasgupta, Xiang Lorraine Li, and Andrew McCallum. 2021a. Probabilistic box embeddings for uncertain knowledge graph reasoning. arXiv preprint arXiv:2104.04597.

Xuelu Chen, Muhao Chen, Weijia Shi, Yizhou Sun, and Carlo Zaniolo. 2019. Embedding uncertain knowledge graphs. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3363–3370.

Xuelu Chen, Ziniu Hu, and Yizhou Sun. 2022. Fuzzy logic based logical query answering on knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3939–3948. Issue: 4.

Zhu-Mu Chen, Mi-Yen Yeh, and Tei-Wei Kuo. 2021b. Passleaf: A pool-based semi-supervised learning framework for uncertain knowledge graph embedding. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):4019–4026.

Narendra Choudhary, Nikhil Rao, Sumeet Katariya, Karthik Subbian, and Chandan Reddy. 2021. Probabilistic entity representation model for reasoning over knowledge graphs. Advances in Neural Information Processing Systems, 34:23440–23451.

Weizhi Fei, Xueyan Niu, Guoqing Xie, Yingqing Liu, Bo Bai, and Wei Han. 2025a. Efficient prompt compression with evaluator heads for long-context transformer inference. arXiv preprint arXiv:2501.12959.

Weizhi Fei, Xueyan Niu, Guoqing Xie, Yanhua Zhang, Bo Bai, Lei Deng, and Wei Han. 2024a. Retrieval meets reasoning: Dynamic in-context editing for long-text understanding. arXiv preprint arXiv:2406.12331.

Weizhi Fei, Xueyan Niu, Pingyi Zhou, Lu Hou, Bo Bai, Lei Deng, and Wei Han. 2024b. Extending context window of large language models via semantic compression. In ACL (Findings).

Weizhi Fei, Zihao Wang, Shukai Zhao, Wei Zhang, Yangqiu Song, et al. 2025b. Efficient and scalable neural symbolic search for knowledge graph complex query answering. arXiv preprint arXiv:2505.08155.

Will Hamilton, Payal Bajaj, Marinka Zitnik, Dan Jurafsky, and Jure Leskovec. 2018. Embedding logical queries on knowledge graphs. Advances in neural information processing systems, 31.

Olaf Hartig and Ralf Heese. 2007. The sparql query graph model for query optimization. In European Semantic Web Conference, pages 564–578. Springer.

Qi Hu, Haoran Li, Jiaxin Bai, Zihao Wang, and Yangqiu Song. 2024. Privacy-preserved neural graph databases. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1108–1118.

Shuang Liang. 2023. Knowledge graph embedding based on graph neural network. In 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 3908–3912.

Lihui Liu, Boxin Du, Heng Ji, ChengXiang Zhai, and Hanghang Tong. 2021. Neural-Answering Logical Queries on Knowledge Graphs. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1087–1097.

Lihui Liu, Zihao Wang, Jiaxin Bai, Yangqiu Song, and Hanghang Tong. 2024a. New frontiers of knowledge graph reasoning: Recent advances and future trends. In Companion Proceedings of the ACM Web Conference 2024, pages 1294–1297.

Lihui Liu, Zihao Wang, and Hanghang Tong. 2024b. Neural-symbolic reasoning over knowledge graphs: A survey from a query perspective. arXiv preprint arXiv:2412.10390.
Tie-Yan Liu et al. 2009. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval, 3(3):225–331.

Francois Luus, Prithviraj Sen, Pavan Kapanipathi, Ryan Riegel, Ndihwuho Makondo, Thabang Lebese, and Alexander Gray. 2021. Logic embeddings for complex query answering. arXiv preprint arXiv:2103.00418.

George A. Miller. 1995. WordNet: A lexical database for English. Communications of the ACM, 38(11):39–41.

Sumit Pai and Luca Costabello. 2021. Learning embeddings from knowledge graphs with numeric edge attributes. arXiv preprint arXiv:2105.08683.

Raymond Reiter. 1981. On closed world data bases. In Readings in artificial intelligence, pages 119–140. Elsevier.

Raymond Reiter. 1986. A sound and sometimes complete query evaluation algorithm for relational databases with null values. Journal of the ACM (JACM), 33(2):349–370.

H Ren, W Hu, and J Leskovec. 2020. Query2box: Reasoning Over Knowledge Graphs In Vector Space Using Box Embeddings. In International Conference on Learning Representations (ICLR).

Hongyu Ren, Mikhail Galkin, Michael Cochez, Zhaocheng Zhu, and Jure Leskovec. 2023. Neural Graph Reasoning: Complex Logical Query Answering Meets Graph Databases. arXiv preprint ArXiv:2303.14617 [cs].

Hongyu Ren and Jure Leskovec. 2020. Beta embeddings for multi-hop logical reasoning in knowledge graphs. Advances in Neural Information Processing Systems, 33:19716–19726.

Kaspar Riesen, Xiaoyi Jiang, and Horst Bunke. 2010. Exact and inexact graph matching: Methodology and applications. Managing and mining graph data, pages 217–247.

Francesca Rossi, Peter van Beek, and Toby Walsh. 2006. Handbook of Constraint Programming. Elsevier Science Inc., USA.

Thomas Schiex. 1992. Possibilistic constraint satisfaction problems or “how to handle soft constraints?””. In Uncertainty in Artificial Intelligence, pages 268–275. Elsevier.

Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 4444–4451. AAAI Press.

Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: a core of semantic knowledge. In Proceedings of the 16th international conference on World Wide Web, pages 697–706.

Dan Suciu, Dan Olteanu, Christopher Ré, and Christoph Koch. 2022. Probabilistic databases. Springer Nature.

Damian Szkarczyk, Rebecca Kirsch, Mikaela Koutrouli, Katerina Nastou, Farrokh Mehryary, Radja Hachilif, Annika L Gable, Tao Fang, Nadezhda T Doncheva, Sampo Pyysalo, et al. 2023. The string database in 2023: protein–protein association networks and functional enrichment analyses for any sequenced genome of interest. Nucleic acids research, 51(D1):D638–D646.

Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International conference on machine learning, pages 2071–2080. PMLR.

Hong Ting Tsang, Zihao Wang, and Yangqiu Song. 2025. Transformers for complex query answering over knowledge hypergraphs. arXiv preprint arXiv:2504.16537.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85. Publisher: ACM New York, NY, USA.

Zihao Wang, Weizhi Fei, Hang Yin, Yangqiu Song, Ginny Y Wong, and Simon See. 2023a. Wasserstein-Fisher-Rao Embedding: Logical Query Embeddings with Local Comparison and Global Transport. arXiv preprint arXiv:2305.04034.

Zihao Wang, Yangqiu Song, Ginny Wong, and Simon See. 2023b. Logical Message Passing Networks with One-hop Inference on Atomic Formulas. In The Eleventh International Conference on Learning Representations.

Zihao Wang, Hang Yin, and Yangqiu Song. 2021. Benchmarking the Combinatorial Generalizability of Complex Query Answering on Knowledge Graphs. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1.

Yao Xu, Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Guang Liu, Jun Zhao, and Kang Liu. 2024. Generate-on-graph: Treat llm as both agent and kg for incomplete knowledge graph question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18410–18430.

Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Dong Yang, Peijun Qing, Yang Li, Haonan Lu, and Xiaodong Lin. 2022a. GammaE: Gamma Embeddings for Logical Queries on Knowledge Graphs. arXiv preprint. ArXiv:2210.15578 [cs].

Haotong Yang, Zhouchen Lin, and Muhan Zhang. 2022b. Rethinking knowledge graph evaluation under the open-world assumption. Advances in Neural Information Processing Systems, 35:8374–8385.

Hang Yin, Zihao Wang, and Yangqiu Song. 2024. Rethinking existential first order queries and their inference on knowledge graphs. In The Twelfth International Conference on Learning Representations.

Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song, and Cane Wing-Ki Leung. 2020. Aser: A large-scale eventuality knowledge graph. In Proceedings of the web conference 2020, pages 201–211.

Zhanqiu Zhang, Jie Wang, Jiajun Chen, Shuiwang Ji, and Feng Wu. 2021. Cone: Cone embeddings for multi-hop reasoning over knowledge graphs. Advances in Neural Information Processing Systems, 34:19172–19183.

Tianshi Zheng, Jiaxin Bai, Yicheng Wang, Tianqing Fang, Yue Guo, Yauwai Yim, and Yangqiu Song. 2024. Clr-fact: Evaluating the complex logical reasoning capability of large language models over factual knowledge. arXiv preprint arXiv:2407.20564.

Tianshi Zheng, Jiazheng Wang, Zhao Wang, Jiaxin Bai, Hang Yin, Zheye Deng, Yangqiu Song, and Jianxin Li. 2025. Enhancing transformers for generalizable first-order logical entailment. arXiv preprint arXiv:2501.00759.

Zhaocheng Zhu, Mikhail Galkin, Zuobai Zhang, and Jian Tang. 2022. Neural-Symbolic Models for Logical Queries on Knowledge Graphs. arXiv preprint arXiv:2205.10128.
Appendix

A  Related Work

A.1  Complex logical queries

Answering complex logical queries over knowledge graphs is naturally extended from link prediction and aims to handle queries with complex conditions beyond simple link queries. This task gradually grows by extending the scope of complex logical queries, ranging from conjunctive queries (Hamilton et al., 2018) to Existential Positive First-Order (EPFO) queries (Ren et al., 2020), Existential First-Order (EFO) queries (Ren and Leskovec, 2020), real Existential First-Order queries (Yin et al., 2024; Zheng et al., 2025). The primary method is query embedding, which maps queries and entities to a low-dimensional space. The form of embedding has been well investigated, such as vectors (Hamilton et al., 2018; Chen et al., 2022; Bai et al., 2022), geometric regions (Ren et al., 2020; Zhang et al., 2021), and probabilistic distributions (Ren and Leskovec, 2020; Choudhary et al., 2021; Yang et al., 2022a; Wang et al., 2023a). These methods not only explore knowledge graphs embedding but also leverage neural logical operators to generate the embedding of complex logical queries.

There are also neural-symbolic models to answer complex logical queries. Gradient optimization techniques were employed to estimate the embedding existential variables (Amayuelas et al., 2021; Arakelyan et al., 2023). Graph neural network (Zhu et al., 2022) was adapted to execute relational projects and use logical operations over fuzzy sets to deal with more complex queries. Efficient search algorithms based on link predictor over knowledge graphs were presented (Yin et al., 2024; Bai et al., 2023b; Fei et al., 2025a). While symbolic methods demonstrate good performance and offer interpretability for intermediate variables, they often struggle to scale with larger graphs due to their high computational complexity.

In addition to uncertain knowledge graphs (KGs), research is also exploring hypergraphs (Tsang et al., 2025) and event-based graphs (Bai et al., 2023a). Due to the strong knowledge representation and logical reasoning of large language models (Fei et al., 2024b,a, 2025b), several studies utilize LLMs to tackle complex logical queries (Xu et al., 2024; Zheng et al., 2024). Numerous other models and datasets have been proposed to enhance query performance and introduce additional features; for a comprehensive overview, see the survey by (Ren et al., 2023; Bai et al., 2025; Liu et al., 2024a,b).

A.2  Uncertain knowledge graph embedding

Uncertain knowledge graph embedding methods aim to map entities and relations into low-dimensional space, enabling the prediction of unknown link information along with confidence values. There are two primary research directions in this field.

The first line of research focuses on predicting the confidence score of uncertain relation facts. UKGE (Chen et al., 2019) was the pioneering effort to model triple plausibility as the activated product of these embedding vectors. UKGE incorporates soft probabilistic logic rules to provide the plausibility of unseen facts. Building upon this, BEUrRE (Chen et al., 2021a) utilizes complex geometric boxes with probabilistic semantics to represent entities and achieve better performance. Semi-supervised learning was applied (Chen et al., 2021b) to predict the associated confidence scores of positive and negative samples. And graph neural networks were used (Liang, 2023) to represent and predict uncertain knowledge graphs.

The other line of research aims to address link prediction on uncertain knowledge graphs by fitting the likelihood of uncertain facts. To adjust the similar task, FocusE (Pai and Costabello, 2021) was introduced, an additional layer to the knowledge graph embedding architecture. They provide variants of classical embedding methods such as TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), and ComplEx (Trouillon et al., 2016).

B  Ablation study: The impact of soft requirements

The two parameters, \( \alpha \) and \( \beta \) play a crucial role in controlling soft constraints, thus we construct settings with varying values. Specifically, we select “zero”(Z) and “random”(R) for \( \alpha \), and “equal”(E) and
Table 5: The mean NDCG of varing \( \alpha \) and \( \beta \).

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Train</th>
    <th colspan="4">Test</th>
    <th rowspan="2">AVG.</th>
  </tr>
  <tr>
    <th>Z+E</th>
    <th>Z+R</th>
    <th>N+E</th>
    <th>N+R</th>
  </tr>
  <tr>
    <td rowspan="4">ConE+NE</td>
    <td>Z+E</td>
    <td><b>39.3</b></td>
    <td>35.6</td>
    <td>31.9</td>
    <td>31.5</td>
    <td>34.6</td>
  </tr>
  <tr>
    <td>Z+R</td>
    <td>39.0</td>
    <td><b>42.2</b></td>
    <td>27.8</td>
    <td>28.7</td>
    <td>34.4</td>
  </tr>
  <tr>
    <td>N+E</td>
    <td>26.6</td>
    <td>23.6</td>
    <td>46.3</td>
    <td>44.1</td>
    <td>35.2</td>
  </tr>
  <tr>
    <td>N+R</td>
    <td>14.0</td>
    <td>7.5</td>
    <td>43.5</td>
    <td>42.2</td>
    <td>26.8</td>
  </tr>
  <tr>
    <td>SRC</td>
    <td>-</td>
    <td>34.3</td>
    <td>37.2</td>
    <td><b>46.7</b></td>
    <td><b>45.8</b></td>
    <td><b>41.0</b></td>
  </tr>
</table>

“random”(R) for \( \beta \), as explained in Section 5.3. Moreover, we sample 12 query types from O*NET20k KG. The detailed construction is in Appendix L. For ConE, we train it on each setting and test it across all settings. As for SRC, we directly test it on all settings.

The results in Table 5 demonstrate that SRC outperforms ConE trained on four different settings in terms of average scores. In the “Z+E” and “Z+R” settings, although ConE achieves higher scores when trained and tested within the same setting, its performance considerably declines when generalizing to other settings. Our method SRC exhibits consistent performance across various settings due to its strong generalization by theoretical foundations in Section 4.2.

C Qualitative analysis

We select the following query for the LLM to conduct a qualitative analysis: “ Soft query: (bread, is related to, \( f_1 \), 0.3, 0.9) ∧ (rice, is related to, \( f_1 \), 0.7, 0.3). Four candidate entities: basic, wheat, white, and food.” Among the four candidates, basic and wheat are weakly related to the two soft constraints. While white and food exhibit higher relevance, the constraints related to bread carry greater importance. Consequently, the final answer is bread. However, some LLMs may often select rice due to a tendency to hallucinate the concept of white bread. Such errors highlight the challenges in accurately interpreting soft constraints and the need for more robust mechanisms to handle nuanced queries.

D Logical queries on knowledge graphs

Definition 9 (Knowledge graphs). Let \( \mathcal{E} \) be the set of entities and \( \mathcal{R} \) be the set of relations. A knowledge graph is a set of triples \( \mathcal{G} = \{ (s_i, r_i, o_i) \} \), where \( s_i, o_i \in \mathcal{E} \) are entities and \( r_i \in \mathcal{E} \) is relation.

The fundamental challenge of knowledge graphs lies in dealing with the Open World Assumption (OWA). Unlike the Closed World Assumption (CWA), which considers only observed triples as facts, OWA acknowledges that unobserved triples may also be valid.

The study of logical queries on KG considers the Existential First-Order (EFO) queries, usually with one free variable (Ren and Leskovec, 2020; Wang et al., 2021; Yin et al., 2024).

Definition 10 (Syntax of existential first-order queries). The disjunctive normal form of an existential first-order query \( \Gamma \) is:

\[
\Gamma(y) = \gamma_1(y) \lor \cdots \lor \gamma_q(y),
\]

where \( y \) is the variable to be answered. Each \( \gamma_i(y) \) is a conjunctive query that is expanded as

\[
\gamma_i(y) = \exists x_1, \ldots, x_n.a_{i1} \land \cdots \land a_{im_i}, i = 1, ..., q,
\]

where \( x_1, \ldots, x_n \) are existentially quantified variables, each \( a_{ij} = r(h, t) \) or \( a_{ij} = \neg r(h, t), j = 1, ..., m_i \) is an atomic query, \( r \) is the relation, \( h \) and \( t \) are either an entity in \( \mathcal{E} \) or a variable in \( \{ y, x_1, ..., x_n \} \).

Definition 11. \( \Gamma(s/y) \) denotes the substitution of the entity \( s \) for the variable \( y \).

When all free variables are substituted, a query \( \Gamma(y) \) is transformed into a sentence \( \Gamma(s/y) \). Given \( \mathcal{G} \), answering a query \( \Gamma(y) \) means finding all substitutions, such that the sentence \( \Gamma(s/y) \) is entailed by \( \mathcal{G} \), i.e., \( \mathcal{G} \models \Gamma(s/y) \). The answer set is defined as
Definition 12 (The Answer Set of first-order Query). The answer set of an first-order query is defined by

\[
\mathcal{A}[\Phi(y)] = \{a \in \mathcal{E}|\ \Phi(a) \text{ is True}\}
\]

The answers in the answer set derived from \( G_{train} \) is easy answers. hard answers are the answers in the set difference between the answers from \( G_{valid} \) and \( G_{train} \) (Wang et al., 2021; Ren et al., 2023). The traditional graph-matching methods can not find the answers introduced by new facts (Riesen et al., 2010). Thus, we should develop new methods.

D.1 ML-based method for logical queries on KG

Recent works are dedicated to introducing ML methods, i.e., knowledge graph embeddings, to generalize from \( \mathcal{G} \) to \( \hat{\mathcal{G}} \), so that it approximates \( \mathcal{G}_{test} \).

Query Embeddings (QE). Query embedding methods generally map the query \( \Gamma \) as an embedding (Ren and Leskovec, 2020; Liu et al., 2021; Wang et al., 2023b). One dominant way is to “translate” the procedure of solving logical formulas in Equation (11) into the set operations, such as set projection, intersection, union, and complement. Then, the neural networks are designed to model such set operations in the embedding space. We can see that direct modeling of set operations is incompatible with the concept of necessity and importance introduced in Section 3.

Inference methods. Other methods solve the open world query answering methods in a two-step approach (Arakelyan et al., 2021; Bai et al., 2023b; Yin et al., 2024). In the first step, the pre-trained knowledge graph embedding estimates the \( \mathcal{G}_{test} \). Then, the answers are derived by the fuzzy logic inference or optimization. These methods rely on the standard logic calculation and cannot be directly applied to our SQUK setting introduced in Section 3.

E Connection with Open World Assumption

Evaluating queries over deductive question-answering systems generally follows either the closed-world assumption (CWA) or the open-world assumption (OWA) (Reiter, 1981). Under CWA, only known facts are considered true, whereas OWA assumes that the absence of knowledge does not imply falsity (Reiter, 1986). Since knowledge graphs (KGs) are often incomplete, knowledge graph completion (KGC) has been proposed to address this challenge. In KGC, the observed knowledge in KGs is divided into three nested subsets: \( G_{train} \subset G_{valid} \subset G_{test} \). The KGC model is trained on the \( G_{train} \) subset and evaluated on the \( G_{valid} \) subset to assess its performance. For complex logical query answering, the evaluation considers "hard answers," defined as the set difference between the answers from \( G_{valid} \) and \( G_{train} \) (Wang et al., 2021; Ren et al., 2023). This approach assesses the model’s ability to handle incomplete knowledge and make inferences beyond the observed facts in the training set. The evaluation of complex logical query answering extends beyond CWA but does not fully align with OWA. The same applies to the evaluation of soft query answering. Evaluating under the Open-World Assumption represents a promising avenue for future work (Yang et al., 2022b).

F Constraint Satisfaction Problem and Soft Constraint Satisfaction Problems

Constraint Satisfaction Problems (CSP) is a mathematical question defined as a set of objects whose state must satisfy several constraints or limitations. Each instance of it can be represented as a triple \( (Z, D, C) \), where \( Z = (z_1, \cdots, z_n) \) is a finite tuple of \( n \) variables, \( D = (D_1, \cdots, D_n) \) is the tuple of the domains corresponding to variables in \( Z \), and \( C = \{(C^1_1, C^2_1), \cdots, (C^t_1, C^2_t)\} \) is the finite set of \( t \) constraints. \( D_i \) is the domain of \( z_i \), and \( C^1_i \subset Z \) is the scope of the i-th constraint and \( C^2_i \) specifies how the assignments allowed by this constraint. In general definitions, the constraints in classical Constraint Satisfaction Problems are hard, meaning that none of them can be violated.

Many problems can be viewed as CSP, which include workforce scheduling and the toy 8-queens problem. Conjunctive queries are a special case to be reduced as CSP under open-world assumptions if we set the constant variable’s domain as itself, set the domain of the existential variable and free variable as the entity set \( \mathcal{E} \) of knowledge graphs, and treat atomic formula or its negation as binary constraint by knowledge graph.
Table 6: Different specific soft CSP frameworks modeled as c-semirings.

<table>
  <tr>
    <th>Semiring</th>
    <th>\( E \)</th>
    <th>\( \times_s \)</th>
    <th>\( +_s \)</th>
  </tr>
  <tr>
    <td>Classical</td>
    <td>\( \{T, F\} \)</td>
    <td>\( \land \)</td>
    <td>\( \lor \)</td>
  </tr>
  <tr>
    <td>Fuzzy</td>
    <td>\( [0, 1] \)</td>
    <td>\( min \)</td>
    <td>\( max \)</td>
  </tr>
  <tr>
    <td>k-weighted</td>
    <td>\( \{0, \ldots, k\} \)</td>
    <td>\( + \)</td>
    <td>\( min \)</td>
  </tr>
  <tr>
    <td>Probabilistic</td>
    <td>\( [0, 1] \)</td>
    <td>\( xy \)</td>
    <td>\( max \)</td>
  </tr>
  <tr>
    <td>Valued</td>
    <td>\( E \)</td>
    <td>\( \oplus \)</td>
    <td>\( min_v \)</td>
  </tr>
</table>

Soft Constraint Satisfaction Problems

Though CSP is a very powerful formulation, it fails when real-life problems need to describe the preference of constraint. It usually returns null answers for problems with many constraints, which are called over-constraints. To tackle the above weakness, many versions of Soft Constraint Satisfaction Problems (SCSP) are developed, such as fuzzy CSP, weighted CSP, and probabilistic CSP, which all follow a common semiring structure, where two semiring operations \( \times_s, +_s \) are utilized to model constraint projection and combination respectively. Based on this theoretical background, we propose soft queries based on SCSP. The proposed soft queries will have the advantages of SCSPs and can handle the numeric facts representing uncertainty.

G  Details of Implementation

Our experiments are run on the Nvidia V100-32G.

G.1 adaptive scoring

Let \( E_s, E_r, \) and \( E_t \) be the embedding vectors of entity \( s \), relation \( r \), and entity \( t \) respectively. We parameterize the adaptive scoring calibration using the following learnable affine transformations:

\[
\rho_\theta(s, r, o) = W_1^1 E_s + b_1^1 + W_2^1 E_r + b_2^1 + W_3^1 E_t + b_3^1,
\]
\[
\lambda_\theta(s, r, o) = W_1^2 E_s + b_1^2 + W_2^2 E_r + b_2^2 + W_3^2 E_t + b_3^2,
\]

where \( \{W_j^i, b_j^i\} \) for \( 1 \leq i \leq 2 \) and \( 1 \leq j \leq 3 \) are the learnable parameters.

G.2 Uncertain knowledge graph embedding

We reproduce previous results (Chen et al., 2019, 2021a) and use the same embedding dimension. We search the other parameters including the learning rate from \( \{1e - 3, 5e - 4, 1e - 4\} \) and regularization term \( \lambda \) from \( \{0.1, 0.01, 0.05\} \).

G.3 Query embedding with number embedding

We follow the same hyperparameter of origin paper (Luus et al., 2021; Zhang et al., 2021) but search the learning rate and margin. The embedding dimension of number embedding is the same as the dimension of entity embedding.

G.4 Two strategies of calibration

For learning strategy, we search learning rate from \( \{5e - 4, 5e - 5, 1e - 5\} \). For Debiasing strategy, we search \( \epsilon \) from \( \{0.05, 0.1, 0.15\} \).

G.5 Running times

When inferring simple queries such as 1p, 2i, and 2in, our method can be parallelized to dynamically compute truth values, resulting in inference times comparable to LMPNN and LogicE. For more complex queries like 2p and 2im, we need to perform serial inference using sparse relation matrices. The inference time for 2,000 2p queries on CN15K is 150 seconds, whereas LogicE and LMPNN take about 10 seconds. Although our approach requires more time, it yields better results than these methods.
Figure 3: A toy model to present the process of SRC algorithm.

H  Details of Soft Reasoning with Calibrated Confidence Values

H.1  Uncertain value definition

By indexing all entities and relations, we represent s, r, and o as integers. Confidence score prediction over uncertain knowledge graphs can be conceptualized by the neural link predictor as \( |\mathcal{R}| \) relation matrices \( \hat{P}_r \in \mathbb{R}^{|\mathcal{E}| \times |\mathcal{E}|} \), where \( \hat{P}_r(s, o) \) is the predicted score of fact triple \( (s, r, o) \) and n is the number of entities. We denote \( U_r(s) \) as a vector formed by the elements of the s-th row. The symbol \( + \) also denotes element-wise plus operation when used in vector-vector or matrix-matrix operations. We also define two new plus operations in matrix-vector operations.

Definition 13. Given a matrix \( M_1 \in \mathbb{R}^{|\mathcal{E}| \times |\mathcal{E}|} \) and a vector \( b \in \mathbb{R}^{|\mathcal{E}|} \), We define two new addition operations: column-wise addition and row-wise addition as following:

\[
M_2 = M_1 +_r b, \quad M_2(s, o) = M_1(s, o) + b(s),
\]
\[
M_2 = M_1 +_c b, \quad M_2(s, o) = M_1(s, o) + b(o).
\]

Definition 14 (Membership function). Given a soft query and a variable x, \( \mu(x, C_x) \) is a membership function to check the current confidence value, where \( \tilde{U}(\mu(s/x, C_x)) = C_x(s) \).

H.2  Details of the inference of SRC

Since the effect \( \alpha \) and \( \beta \) are equivalent to modifying the uncertain values, we will focus on explaining how the method works in the basic scenario where \( \alpha = 0.0 \) and \( \beta = 1.0 \). Our goal is to infer the utility vector \( \tilde{U}[\phi(y)] \) by estimating the confidence value \( \tilde{U}[\phi](o) = [\hat{P} \models_s \phi(o/y)] \), for all \( o \in \mathcal{E} \). In the following content, we will cut the query graph step by step while recording any lost information on the remaining nodes. After removing nodes from the soft query \( \phi \), we denote the remaining sub-query graph as \( \psi \).

H.2.1  Step 1. Initialization.

We initialize each variable x as a candidate state vector \( C \) with all zero elements, denoted as \( C = 0 \in \mathbb{R}^{|\mathcal{E}|} \). This vector records the candidates and their corresponding values. Throughout the algorithmic process, the vectors \( C \) are updated iteratively, ultimately yielding the final answers represented by the resulting vector \( C_y \).

H.2.2  Step 2. Remove constant nodes.

For constant nodes in a query graph, we can easily remove their whole edges and update the information to connected nodes by the following lemma.

Lemma 3. For the constant nodes in a soft query, there exists an \( O(|\mathcal{E}|) \) transformation \( T_c \) to remove them.

Proof. Without loss of generality, we consider the situation that a constant node with entity e connects an existential variable x that there is only one positive edge from e to x, one negative edge from e to x, and
one positive edge from \( x \) to \( e \). To simplify, we also denote \( e \) as the related grounded entity.

\[
\hat{U}[\phi](o) = \hat{U}(\exists x.\mu(x, C_x) \land r_1(e, x) \land \neg r_2(e, x) \land r_3(x, e) \land \psi(o/y)) \\
= \max_{s \in \mathcal{E}}[C_x(s) + \hat{P}_{r_1}(e, s) + (1 - \hat{P}_{r_2})(e, s) \\
+ \hat{P}_{r_3}^T(e, s) + \hat{U}(\psi(o/y; s/x))] \\
= \max_{s \in \mathcal{E}}[C'_x(s) + \hat{U}(\psi(o/y; s/x))],
\]

where \( C'_x = C_x + \hat{P}_{r_1}(e) + (1 - \hat{P}_{r_2})(e) + \hat{P}_{r_3}^T(e) \) is updated candidate vector and \( 1 \in \mathbb{R}^{|\mathcal{E}| \times |\mathcal{E}|} \) is all one matrix. The situation where a constant node connects a free variable node is similar and even easier to handle. \( \square \)

In the above derivation, we retain the value of answers by updating the candidate state vector of an existential variable.

H.2.3 Step 3. Remove self-loop edges.

Lemma 4. For a soft query having self-loop edges, there exists an \( O(|\mathcal{E}|) \) transformation \( T_c \) to remove self-loop edges.

Proof. Without loss of generality, we consider the situation in which an existential variable \( x \) contains one positive loop.

\[
\hat{U}[\phi](o) = \hat{U}(\exists x.\mu(x, C_x) \land r(x, x) \land \psi(o/y; x)) \\
= \max_{s \in \mathcal{E}}[C_x(s) + \hat{P}_r(s, s) + \hat{U}(\psi(o/y; s/x))] \\
= \max_{s \in \mathcal{E}}[C'_x(s) + \hat{U}(\psi(o/y; s/x))],
\]

where \( C'_x = C_x + diag(\hat{P}_r) \) and \( diag(\hat{P}_r) \) is a vector formed by the diagonal elements of matrix \( \hat{P}_r \). \( \square \)

Previous research (Yin et al., 2024) has demonstrated the difficulty of sampling high-quality self-loop queries due to the rarity of self-loop relations in real-life knowledge graphs. Similarly, in our specific uncertain knowledge graph, it is challenging to sample meaningful queries.

H.2.4 Step 4. Remove leaf nodes.

The leaf node \( u \), which is only connected to one other node \( v \) in the soft query graph. After removing the constant nodes, if the query graph contains no circles, we can get the utility vector by removing the leaf nodes step by step. Next, we present how to handle leaf nodes by the three lemmas.

Lemma 5. If the leaf node \( u \) is an existential variable \( x \) and \( v \) is the free variable \( y \), there exists an \( O(|\mathcal{E}|^2) \) transformation \( T_l \) to shrink its graph.

\[
\hat{U}[\phi](o) = \hat{U}(\exists x.\mu(x, C_x) \land r(x, o) \land \mu(o, C_y) \land \psi(o/y)) \\
= \max_{s \in \mathcal{E}}[C_x(s) + \hat{P}_r(s, o) + C_y(o) + \hat{U}(\psi(o/y))] \\
= \max_{s \in \mathcal{E}}[C_x(s) + \hat{P}_r(s, o) + C_y(o)] + \hat{U}(\psi(o/y)) \\
= C'_y(o) + \hat{U}(\psi(o/y)),
\]

where \( C'_y(o) = \max_{s \in \mathcal{E}} M_1(s, o) \) and \( M_1 = [(\hat{P}_r + c\ C_y) + r\ C_x] \).

Lemma 6. If the leaf node \( u \) is a free variable \( y \) and \( v \) is the existential variable \( x \), we can first solve the subgraph obtained by removing \( u \), and then remove \( v \).
Proof. Denote \( \psi \) is the subgraph obtained by removing \( u \), we treat \( y \) as the free variable to get its utility vector \( \hat{U}(\psi(y)) \). Then we can remove \( u \) by the following.

\[
\begin{align*}
\hat{U}[\phi](o) &= \hat{U}(\mu(o, C_y) \land [\exists x.r(x, o) \land \psi(x)]) \\
&= C_y(o) + \hat{U}(\exists x.r(x, o) \land \psi(x)) \\
&= C_y(o) + \max_{s \in \mathcal{E}} [\hat{P}_r(s, o) + \hat{U}(\psi(o))] \\
&= C_y(o) + \max_{s \in \mathcal{E}} M_2(s, o),
\end{align*}
\]

where \( M_2 = \hat{P}_r +_c \hat{U}[\psi] \).

Lemma 7. If the leaf node \( u \) and its connected node \( v \) both are the existential variable, we can remove the leaf node \( u \) when the existential quantifier is maximization.

Proof. It will be difficult when trying to cut the leaf node \( x_1 \),

\[
\begin{align*}
\hat{U}[\phi](o) &= \hat{U}(\exists x_1, x_2.\mu(x_1, C_{x_1}) \land r(x_1, x_2) \land \mu(x_2, C_{x_2}) \land \psi(o)) \\
&= \max_{s_1 \in \mathcal{E}, s_2 \in \mathcal{E}} [C_{x_1}(s_1) + \hat{P}_r(s_1, s_2) + C_{x_2}(s_2) + \hat{U}(\psi(o; s_2/x_2))] \\
&= \max_{s_2 \in \mathcal{E}} \max_{s_1 \in \mathcal{E}} [C_{x_1}(s_1) + \hat{P}_r(s_1, s_2) + C_{x_2}(s_2) + \hat{U}(\psi(o; s_2/x_2))].
\end{align*}
\]

While the existential quantifier is maximization, it is noteworthy that for any \( s_1, s_2, o \in \mathcal{E} \),

\[
\max_{s_1 \in \mathcal{E}} [M_3(s_1, s_2) + \hat{U}(\psi(o; s_2/x_2))] = \max_{s_1 \in \mathcal{E}} [M_3(s_1, s_2)] + \hat{U}(\psi(o; s_2/x_2)) \\
= C'_{x_2}(s_2) + \hat{U}(\psi(o; s_2/x_2)),
\]

where \( M_3 = (\hat{P}_r +_r C_{x_1}) +_c C_{x_2} \) and \( C'_{x_2}(s_2) = \max_{s_1 \in \mathcal{E}} [M_3(s_1, s_2)] \).

Therefore, we can remove \( x_1 \) by updating \( x_2 \) as follows,

\[
\hat{U}[\phi](o) = \max_{s_2 \in \mathcal{E}} [C'_{x_2}(s_2) + \hat{U}(\psi(o/y; s_2/x_2))].
\]

Combining the above three lemmas, we can step by step find a leaf node and remove it when the query graph has no cycles.

Lemma 8. If the soft query contains no circles, we can get the utility vector by removing leaf nodes when the existential quantifier is maximization.

H.2.5 Step 5. Enumerate on the cycle.

To the best of our knowledge, the only precise approach for addressing cyclic queries is performing enumeration over one existential node involved in the cycle, which reads \( \hat{U}(\exists x.\phi(o/y; x)) = \max_{s \in \mathcal{E}} \hat{U}(\phi(o/y; s/x)) \). Then, we apply Step 4 to remove this fixed existential variable since this variable is equivalent to the constant variable. The query graph breaks this cycle and becomes smaller. The remaining query can be solved by applying Step 4. When solving cyclic queries, the time complexity of this algorithm is exponential.

H.2.6 Step 6. Getting the utility vector.

Following the aforementioned steps, the query graph will only contain the free node \( y \), resulting in the formula \( \mu(y, C_y) \). By definition, the desired utility vector will be \( C_y \), which provides the confidence values of all the candidate entities.
Table 7: The results of answering complex soft queries with different backbone models.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Metrics</th>
    <th colspan="6">IP</th>
    <th colspan="6">2P</th>
    <th colspan="6">2I</th>
    <th colspan="6">2IL</th>
    <th colspan="6">2M</th>
    <th colspan="6">2U</th>
    <th colspan="6">3IN</th>
    <th colspan="6">IP</th>
    <th colspan="6">IM</th>
    <th colspan="6">INP</th>
    <th colspan="6">UP</th>
    <th colspan="6">AVG</th>
  </tr>
  <tr>
    <th>1P</th><th>2P</th><th>2I</th><th>2IN</th><th>2IL</th><th>2M</th><th>2U</th><th>3IN</th><th>IP</th><th>IM</th><th>INP</th><th>UP</th><th>AVG</th>
  </tr>
  <tr>
    <td colspan="34">CN15k</td>
  </tr>
  <tr>
    <td rowspan="4">SRC (UKGE)</td>
    <td>MAP</td><td>20.6</td><td>7.1</td><td>7.9</td><td>14.4</td><td>14.6</td><td>3.3</td><td>14.7</td><td>7.7</td><td>6.2</td><td>3.2</td><td>4.2</td><td>6.5</td><td>9.2</td>
  </tr>
  <tr>
    <td>NDCG</td><td>27.7</td><td>15.4</td><td>10.1</td><td>23.8</td><td>22.6</td><td>7.9</td><td>24.3</td><td>10.4</td><td>11.1</td><td>6.9</td><td>10.5</td><td>14.9</td><td>15.5</td>
  </tr>
  <tr>
    <td>\( \rho \)</td><td>21.9</td><td>6.4</td><td>0.2</td><td>5.9</td><td>14.3</td><td>11.2</td><td>32.0</td><td>-1.5</td><td>-7.0</td><td>8.4</td><td>-2.3</td><td>17.4</td><td>8.9</td>
  </tr>
  <tr>
    <td>\( \tau \)</td><td>15.6</td><td>2.4</td><td>0.0</td><td>2.1</td><td>10.8</td><td>9.2</td><td>25.5</td><td>2.0</td><td>9.7</td><td>4.1</td><td>4.4</td><td>13.0</td><td>5.9</td>
  </tr>
  <tr>
    <td rowspan="4">SRC (BEuRE)</td>
    <td>MAP</td><td>32.2</td><td>11.5</td><td>13.2</td><td>25.9</td><td>29.2</td><td>3.9</td><td>26.6</td><td>13.9</td><td>12.8</td><td>5.3</td><td>8.4</td><td>11.6</td><td>16.2</td>
  </tr>
  <tr>
    <td>NDCG</td><td>41.5</td><td>21.3</td><td>15.7</td><td>37.4</td><td>39.7</td><td>9.7</td><td>40.5</td><td>17.6</td><td>19.0</td><td>10.4</td><td>17.1</td><td>22.2</td><td>24.3</td>
  </tr>
  <tr>
    <td>\( \rho \)</td><td>28.7</td><td>13.9</td><td>2.3</td><td>11.6</td><td>19.0</td><td>17.4</td><td>31.5</td><td>10.4</td><td>21.3</td><td>5.1</td><td>21.5</td><td>11.8</td>
  </tr>
  <tr>
    <td>\( \tau \)</td><td>18.7</td><td>8.7</td><td>-3.5</td><td>7.7</td><td>14.8</td><td>14.7</td><td>27.2</td><td>-1.9</td><td>-3.6</td><td>20.3</td><td>1.8</td><td>16.3</td><td>10.1</td>
  </tr>
  <tr>
    <td colspan="34">PP15k</td>
  </tr>
  <tr>
    <td rowspan="4">SRC (BEuRE)</td>
    <td>MAP</td><td>78.2</td><td>78.4</td><td>72.9</td><td>67.0</td><td>72.0</td><td>52.0</td><td>73.7</td><td>88.6</td><td>76.4</td><td>54.2</td><td>69.2</td><td>67.8</td><td>68.4</td>
  </tr>
  <tr>
    <td>NDCG</td><td>80.7</td><td>78.2</td><td>77.3</td><td>68.7</td><td>78.0</td><td>52.5</td><td>79.9</td><td>82.5</td><td>76.2</td><td>52.8</td><td>69.4</td><td>72.4</td><td>70.7</td>
  </tr>
  <tr>
    <td>\( \rho \)</td><td>77.7</td><td>75.3</td><td>70.9</td><td>59.9</td><td>74.8</td><td>54.3</td><td>78.8</td><td>82.2</td><td>74.6</td><td>54.8</td><td>69.6</td><td>70.7</td><td>66.7</td>
  </tr>
  <tr>
    <td>\( \tau \)</td><td>66.6</td><td>70.9</td><td>49.9</td><td>42.7</td><td>71.0</td><td>42.9</td><td>71.6</td><td>32.7</td><td>65.6</td><td>37.1</td><td>57.2</td><td>70.4</td><td>56.5</td>
  </tr>
  <tr>
    <td rowspan="4">SRC (BEuRE)</td>
    <td>MAP</td><td>71.0</td><td>67.8</td><td>63.3</td><td>56.3</td><td>63.6</td><td>51.8</td><td>66.3</td><td>49.4</td><td>67.7</td><td>52.6</td><td>57.7</td><td>58.8</td><td>60.5</td>
  </tr>
  <tr>
    <td>NDCG</td><td>74.8</td><td>70.5</td><td>68.7</td><td>61.5</td><td>70.0</td><td>49.7</td><td>74.2</td><td>55.5</td><td>68.5</td><td>49.1</td><td>61.8</td><td>66.5</td><td>64.2</td>
  </tr>
  <tr>
    <td>\( \rho \)</td><td>72.7</td><td>76.4</td><td>51.6</td><td>47.1</td><td>73.9</td><td>51.3</td><td>77.3</td><td>39.0</td><td>68.5</td><td>43.2</td><td>63.2</td><td>75.0</td><td>61.6</td>
  </tr>
  <tr>
    <td>\( \tau \)</td><td>59.7</td><td>62.3</td><td>41.8</td><td>37.7</td><td>61.9</td><td>39.4</td><td>64.2</td><td>32.0</td><td>55.8</td><td>32.6</td><td>50.6</td><td>61.5</td><td>50.0</td>
  </tr>
  <tr>
    <td colspan="34">O*NET20k.</td>
  </tr>
  <tr>
    <td rowspan="4">SRC (BEuRE)</td>
    <td>MAP</td><td>24.9</td><td>6.4</td><td>70.6</td><td>26.0</td><td>63.3</td><td>5.6</td><td>32.8</td><td>68.5</td><td>7.7</td><td>7.3</td><td>5.2</td><td>7.7</td><td>27.1</td>
  </tr>
  <tr>
    <td>NDCG</td><td>44.9</td><td>19.5</td><td>80.4</td><td>46.1</td><td>74.0</td><td>17.2</td><td>57.2</td><td>76.3</td><td>19.2</td><td>17.1</td><td>17.7</td><td>24.6</td><td>41.3</td>
  </tr>
  <tr>
    <td>\( \rho \)</td><td>77.9</td><td>64.8</td><td>79.1</td><td>73.7</td><td>79.6</td><td>43.7</td><td>83.3</td><td>69.4</td><td>53.8</td><td>33.3</td><td>58.0</td><td>60.7</td><td>65.3</td>
  </tr>
  <tr>
    <td>\( \tau \)</td><td>72.0</td><td>54.9</td><td>68.6</td><td>67.6</td><td>67.3</td><td>36.9</td><td>76.0</td><td>59.2</td><td>47.6</td><td>29.1</td><td>48.9</td><td>52.4</td><td>57.3</td>
  </tr>
  <tr>
    <td rowspan="4">SRC (BEuRE)</td>
    <td>MAP</td><td>29.0</td><td>8.3</td><td>65.2</td><td>29.6</td><td>55.2</td><td>7.2</td><td>32.3</td><td>62.1</td><td>9.9</td><td>8.8</td><td>6.9</td><td>9.6</td><td>27.1</td>
  </tr>
  <tr>
    <td>NDCG</td><td>53.4</td><td>29.2</td><td>78.8</td><td>52.4</td><td>71.0</td><td>21.5</td><td>60.2</td><td>72.7</td><td>26.5</td><td>20.4</td><td>25.5</td><td>34.1</td><td>45.8</td>
  </tr>
  <tr>
    <td>\( \rho \)</td><td>69.0</td><td>59.1</td><td>78.1</td><td>63.6</td><td>76.4</td><td>40.1</td><td>78.3</td><td>66.8</td><td>47.6</td><td>30.4</td><td>51.2</td><td>57.2</td><td>60.2</td>
  </tr>
  <tr>
    <td>\( \tau \)</td><td>58.4</td><td>47.1</td><td>66.6</td><td>53.6</td><td>62.8</td><td>32.0</td><td>67.4</td><td>55.9</td><td>39.2</td><td>24.9</td><td>40.5</td><td>46.8</td><td>49.9</td>
  </tr>
</table>

I Uncertain Knowledge Graph Embeddings

We introduce the backbone models for uncertain knowledge graph embedding. The results of changing the backbone are presented in Table 7.

UKGE (Chen, 2023) is a vector embedding model designed for uncertain knowledge graphs. It has been tested on three tasks: confidence prediction, relational fact ranking, and relational fact classification. To address the sparsity issue in the graph, UKGE utilizes probabilistic soft logic, allowing for the inclusion of additional unseen relational facts during training.

BEUrRE (Chen et al., 2021a) is a probabilistic box embedding model that has been evaluated on two tasks: confidence prediction and relational fact ranking. This model represents each entity as a box and captures relations between two entities through affine transformations applied to the head and tail entity boxes.

J Float Embedding for query Embedding

To enable query embedding methods to handle soft requirements in soft queries, we employ floating-point encoding to map floating-point numbers into vectors. These vectors are then added to the relation projection in the query embedding method.

J.1 Float embedding

We consider the sinusoidal encoding \( g : \mathbb{R} \to R^d \) introduced in Transformer (Vaswani et al., 2017) and map the values of \( \alpha \) and \( \beta \) into vector embedding, which can be formulated as:

\[
g(v_i) = \begin{cases}
sin(v_i/1000^{i/(2k)}) & i = 2k, \\
cos(v_i/1000^{i/(2k)}) & i = 2k + 1,
\end{cases}
\]

where \( d \) is the embedding dimension.

J.2 Modified relation projection

The query embedding methods usually learn a Multi-Layer Perceptron (MLP) for each relation \( r \), which reads as:

\[
S' = \text{MLP}_r(S),
\]
Figure 4: Query structures of query types. The white, yellow, and red circles represent constant, existential, and free nodes, respectively. The negative atomic formulas are represented by red edges, while atomic formulas are represented by black edges. Like the previous naming convention (Ren and Leskovec, 2020; Yin et al., 2024), we use “P” for projection, “I” for intersection, “N” for negation, “M” for multi-edge, and “L” for existential leaf.

where \( S \) is an embedding. Furthermore, the modified relation projection can be expressed as:

\[
S' = \text{MLP}_r(S + g(\alpha) + g(\beta))
\]

Here, \( g(\alpha) \) and \( g(\beta) \) are the embedding of soft requirements \( \alpha \) and \( \beta \), respectively. By incorporating \( g(\alpha) \) and \( g(\beta) \) into the relation project net, we enhance the representation of relation projection to better capture the soft requirement.

K  Details in the Main Dataset Construction

K.1  Uncertain knowledge graphs

We sample soft queries from three standard uncertain knowledge graphs\footnote{We leave the exploration of N127k (Chen et al., 2019) for future work as it involves an inductive setting where the valid/test graphs contain unseen relations.}, covering diverse domains such as common sense knowledge, bioinformatics, and the employment domain.

**CN15k** (Chen et al., 2019) is a subset of the ConceptNet (Speer et al., 2017), a semantic network aimed at comprehending connections between words and phrases.

**PPI5k** (Chen et al., 2019) is a subset of STRING (Szklarczyk et al., 2023), which illustrates protein-protein association networks collected from organisms. It assigns probabilities to the intersections among proteins.

**O*NET20K** (Pai and Costabello, 2021) is a subset of O*NET, a dataset that describes labeled binary relations between job descriptions and skills. The associated values are to evaluate the importance of the link within the triple.

We present the statistics of these three uncertain knowledge graphs in Table ??.

K.2  Soft requirements

In the main experiment, we aim for the sampled data to allow the machine learning methods to generalize to various scenarios of soft requirements. Therefore, for \( \alpha \) necessity, we employ a hybrid strategy, randomly selecting from four modes (“zero”, “low”, “normal”, “high”) for each query. As for \( \beta \) importance, we utilize a random strategy, randomly choosing a decimal value for different atomic soft formulas in each query.

K.3  Query types

The goal of our proposed dataset is to represent the family of existential first-order soft queries systematically. However, including too many query formulas poses challenges in analyzing and evaluating. We select query graphs including operations as train queries and unitize composed query graphs to evaluate the combinatorial generalization.

For training queries, we choose 1P, 2P, 2I, and 2IN, which include soft operators. Additionally, we select 2P for chain queries (Hamilton et al., 2018), 2M for multi-edge graphs (Yin et al., 2024), and 2IL for graphs containing ungrounded anchors (Yin et al., 2024). More complex soft query graphs can be generated from these basic graphs. We present the statistics of sampled queries in Table 8 and Tabel 9.
Table 8: The statistics of valid/test queries on the main dataset. Different query types have the same number in given uncertain knowledge graph.

<table>
  <tr>
    <th></th>
    <th>KG</th>
    <th>CN15k</th>
    <th>PPI5k</th>
    <th>O*NET20k</th>
  </tr>
  <tr>
    <th>valid</th>
    <td>3,000</td>
    <td>2,000</td>
    <td>2,000</td>
    <td>2,000</td>
  </tr>
  <tr>
    <th>test</th>
    <td>3,000</td>
    <td>2,000</td>
    <td>2,000</td>
    <td>2,000</td>
  </tr>
</table>

Table 9: The statistics of train queries on the main dataset.

<table>
  <tr>
    <th></th>
    <th>KG</th>
    <th>1P</th>
    <th>2P</th>
    <th>2I</th>
    <th>2IN</th>
    <th>2IL</th>
  </tr>
  <tr>
    <th>CN15k</th>
    <td>52887</td>
    <td>52900</td>
    <td>52900</td>
    <td>5300</td>
    <td>5300</td>
    <td></td>
  </tr>
  <tr>
    <th>PPI5k</th>
    <td>9724</td>
    <td>9750</td>
    <td>9754</td>
    <td>1500</td>
    <td>1500</td>
    <td></td>
  </tr>
  <tr>
    <th>PPI5k</th>
    <td>18266</td>
    <td>18300</td>
    <td>18300</td>
    <td>1850</td>
    <td>1850</td>
    <td></td>
  </tr>
</table>

K.4 Evaluation protocol

The open world assumption in uncertain knowledge graphs not only establishes new links between entities but can also potentially refine the values of existing triples. As more observed facts become available, the answers to soft queries not only increase in number but also undergo modifications in terms of their priority. Therefore, to evaluate the relevance judgment, we select several popular metrics commonly used in information retrieval (Liu et al., 2009), including MAP, DCG, NDCG, and Kendall’s tau.

For each \( q \), we denote the set of answers as \( \mathcal{A} \), where \( a_i \in \mathcal{A} \) represents the i-th answer based on its score, and \( r(a), a \in \mathcal{A} \) denotes the predicted ranking of answer \( a \in \mathcal{A} \). Our objective is to focus on the precision of answers and the associated predicted ranking information.

**Mean Average Precision (MAP):** To define MAP, we first introduce Precision at a given position, defined as:

\[
\mathbf{P@k}(q) = |\{a \in \mathcal{A}|r(a) \geq k\}|/k.
\]

(26)

Then, Average Precision is defined as follows:

\[
\mathbf{AP}(q) = (\sum_{k=1}^{|\mathcal{A}|} \mathbf{P@k}(q) \cdot l_k)/|\mathcal{A}|,
\]

(27)

where \( l_k \) is a binary judgment indicating the relevance of the answer at the kth position. Mean Average Precision is the average AP value across all test queries.

**Discounted Cumulative Gain (DCG):** To calculate the DCG, we utilize the Reciprocal Rank as a relative score for the answers:

\[
R(a_i) = 1/r(a_i).
\]

(28)

To incorporate the ranking position, we introduce an explicit position discount factor \( \eta_i \). The DCG is then computed as:

\[
\mathbf{DCG@k}(q) = \sum_{i=1}^k R(a_i)\eta(i),
\]

(29)

where \( \eta(i) \) is commonly expressed as \( \eta(i) = 1/\log_2(i+1) \).

**Normalized Discounted Cumulative Gain (NDCG):** By normalizing the ideal Discounted Cumulative Gain denoted as \( Z_k \), we obtain NDCG:

\[
\mathbf{NDCG@k}(q) = \mathbf{DCG@k}(q)/Z_k.
\]

(30)

**Kendall’s tau:** Kendall’s tau is a statistical measure that quantifies the correspondence between two rankings. Values close to 1 indicate strong agreement, while values close to \(-1\) indicate strong disagreement.
Table 10: The additional results of varying soft requirements

<table>
  <tr>
    <th rowspan="2">Mode Metric</th>
    <th colspan="2">Z+E</th>
    <th colspan="2">Z+R</th>
    <th colspan="2">N+E</th>
    <th colspan="2">N+R</th>
  </tr>
  <tr>
    <th>NDCG</th><th>\( \tau \)</th>
    <th>NDCG</th><th>\( \tau \)</th>
    <th>NDCG</th><th>\( \tau \)</th>
    <th>NDCG</th><th>\( \tau \)</th>
  </tr>
  <tr>
    <td>ConE Z+E</td>
    <td>39.3</td><td>8.0</td>
    <td>35.6</td><td>11.6</td>
    <td>31.9</td><td>30.4</td>
    <td>31.5</td><td>28.9</td>
  </tr>
  <tr>
    <td>ConE Z+R</td>
    <td>39.0</td><td>6.7</td>
    <td>42.2</td><td>14.9</td>
    <td>27.8</td><td>29.0</td>
    <td>28.7</td><td>30.3</td>
  </tr>
  <tr>
    <td>ConE N+E</td>
    <td>26.6</td><td>32.8</td>
    <td>23.6</td><td>27.4</td>
    <td>46.3</td><td>38.7</td>
    <td>44.1</td><td>36.7</td>
  </tr>
  <tr>
    <td>ConE N+R</td>
    <td>14.0</td><td>24.7</td>
    <td>7.5</td><td>14.4</td>
    <td>43.5</td><td>38.7</td>
    <td>42.2</td><td>38.6</td>
  </tr>
  <tr>
    <td>SRC</td>
    <td>34.3</td><td>47.8</td>
    <td>37.2</td><td>41.5</td>
    <td>46.7</td><td>51.8</td>
    <td>45.8</td><td>49.9</td>
  </tr>
</table>

Table 11: The number of annotated queries.

<table>
  <tr>
    <th>Query type</th>
    <th>1P</th>
    <th>2P</th>
    <th>2I</th>
    <th>2IN</th>
    <th>2IL</th>
    <th>2M</th>
    <th>3IN</th>
    <th>IP</th>
    <th>IM</th>
    <th>SUM</th>
  </tr>
  <tr>
    <th>Number</th>
    <td>100</td>
    <td>50</td>
    <td>100</td>
    <td>50</td>
    <td>15</td>
    <td>10</td>
    <td>15</td>
    <td>15</td>
    <td>15</td>
    <td>370</td>
  </tr>
</table>

L  Details in Varying Soft Requirements Setting

In this setting, we aim to test the model’s generalization ability on soft requirements under different strategies. We have chosen a pair of different strategies for each of the two parameters, resulting in a total of four groups. Specifically, we selected "zero" and "normal" for parameter "a," and "equal" and "normal" for parameter "b." Detailed statistics for this setting can be found in the table below. For each group, we follow the procedure and sample train, valid, and test queries. We present the additional results in Table 10.

M  Details of Large Language Model Evaluation Setting

Since the entities and relations in CN15k are English words and phrases, the queries sampled from CN15k can be well understood by LLM. In our evaluation, we manually marked and removed meaningless queries. To facilitate our testing process, we selected only four candidate entities for each query. These four candidate entities have large distinctions in terms of scores. To avoid unexpected situations, we manually checked all chosen queries and confirmed that their correct answers could be selected without ambiguity. We present the total numbers and types of selected queries in Table 11.

We articulate the syntax and semantics of our proposed soft queries by using clear natural language. The provided prompts are presented in the subsequent subsection. Through combining queries with prompts, we enable LLM to select the most appropriate answer.

M.1 Prompt

# Background
Given a soft query containing a free variable f, there are four candidate entities for this variable and you need to choose the best of them to satisfy the soft query most. In order to do so, you can first compute the confidence value to see whether the chosen candidate entity satisfies the soft query after substituting the free variable with a given candidate entity. After computing the confidence values for each corresponding candidate entity, you need to pick the entity that leads to the highest confidence value. The detailed steps are described below:

## Definition of Soft Atomic Constraint:
A soft atomic constraint c, a.k.a. a soft atomic formula or its negation, is in the form of (h, r, t, \alpha, \beta) or \neg (h, r, t, \alpha, \beta).
### Notation Description:
In each constraint, we have four different types of variables.
1. r is a relation;
2. h and t are two terms. Each term represents an entity or a variable whose values are entities. And free variable is a term.
3. \( \alpha \) is called the necessity value, which represents the minimal requirement of the uncertainty degree of this constraint. It can be any decimal between 0.0 and 1.0. If the confidence value is less than the necessity value, the constraint is not satisfied, and thus the final confidence value becomes negative infinity;
4. \( \beta \) represents the priority of this constraint and can be any decimal between 0.0 and 1.0.

### Confidence Value of Soft Constraints V(c):
1. The triple (h,r,t) comes from the relation fact in the knowledge graph. In our setting, the relation fact r(h,t) is not boolean, and it has a confidence value in the range [0.0,1.0] according to its plausibility. When the constraint is negative, you need to first estimate r(h,t)—the confidence value of r(h,t)—and use 1-r(h,t) as the final confidence value for this negative constraint.
2. \( \alpha \) is the threshold in our filter function, working as follows:
\[
f(v, \alpha) = v, \quad \text{if } v \geq \alpha,
\]
\[
-\infty, \quad \text{if } v < \alpha.
\]
3. \( \beta \) is a coefficient.

Thus, the final equation becomes:
\[
V(h, r, t, \alpha, \beta) = \beta \times f(r(h,t), \alpha),
\]
\[
V(\neg (h, r, t, \alpha, \beta)) = \beta \times f(1-r(h,t), \alpha).
\]

## Definition of Conjunctive Queries \( \phi \):
Soft conjunctive queries are composed of soft constraints.

### Notation Description:
Conjunctive query \( \phi = c_1 \land \dots \land c_n \), where \( c_i \) is a soft constraint.

### Confidence Value of Soft Conjunctive Queries V(\( \phi \)) :
First, you need to compute the confidence values of all soft constraints in this soft conjunctive query. Then, you can simply sum up these confidence values as the final confidence value of the soft conjunctive query as follows:
\[
V(\phi) = \sum_i V(c_i).
\]

If the conjunctive query has an existential variable e, you should find an entity to replace it and then compute the confidence value of this query.

# Output Format
Please output your response in the JSON format, where the first element is the best candidate entity among the four options, and the second element is your explanation for your choice.

# Question
Soft query:(h_1,r_1,f,\alpha_1,\beta_1) \land (h_2,r_2,f,\alpha_2,\beta_2)
Four candidate entities: s_1, s_2, s_3, s_4
Please return the best candidate for f1 to satisfy the above soft query.

N  Proof for Error Analysis

N.1  Proof of Theorem 1

Firstly, we give the proof of Theorem 1:

Proof. Consider the atomic query \( \psi = (a, (\alpha, \beta)) \). Firstly, we consider whether the soft atomic query is positive or negative.

Let us assume \( a = r(y, o) \), with \( y \) be the only free variable

Then for arbitrary \( r, h, t, \alpha, \beta \):

\[
\Pr \left( \| \hat{U}[\psi](s) - U[\psi](s) \| > \delta \right) = \Pr \left( \beta \| [\hat{P}(s, r, o)]_\alpha - [\mathcal{P}(s, r, o)]_\alpha \| > \delta \right) \tag{31}
\]
\[
= \Pr \left( \| [\hat{P}(s, r, o)]_\alpha - [\mathcal{P}(s, r, o)]_\alpha \| > \frac{\delta}{\beta} \right) \tag{32}
\]

We note that even if \( a = \neg r(y, o) \), the result is the same:

\[
\Pr \left( \| \hat{U}[\psi](s) - U[\psi](s) \| > \delta \right) = \Pr \left( \beta \| [1 - \hat{P}(s, r, o)]_\alpha + [\mathcal{P}(s, r, o)]_\alpha - 1 \| > \delta \right)
\]
\[
= \Pr \left( \| [\hat{P}(s, r, o)]_\alpha - [\mathcal{P}(s, r, o)]_\alpha \| > \frac{\delta}{\beta} \right)
\]

For convenience, we write \( \hat{x}, x \) as the abbreviation of \( \hat{P}(s, r, o), \mathcal{P}(s, r, o) \), correspondingly, then the initial formula becomes:

\[
\Pr \left( \| \hat{x} - x \| > \frac{\delta}{\beta} \right)
\]
\[
= \Pr \left( \| \hat{x} - x \| > \frac{\delta}{\beta} \mid \hat{x} > \alpha, x > \alpha \right) \Pr(\hat{x}, x > \alpha) + \Pr \left( \| \hat{x} \| > \frac{\delta}{\beta} \mid \hat{x} > \alpha, x < \alpha \right) \Pr(\hat{x} > \alpha, x < \alpha)
\]
\[
+ \Pr \left( \| x \| > \frac{\delta}{\beta} \mid \hat{x} < \alpha, x > \alpha \right) \Pr(\hat{x} < \alpha, x > \alpha) + \Pr \left( \| 0 \| > \frac{\delta}{\beta} \mid \hat{x} < \alpha, x < \alpha \right) \Pr(\hat{x} < \alpha, x < \alpha)
\]
\[
\leq \varepsilon \left( \frac{\delta}{\beta} \right) \Pr(\hat{x} > \alpha, x > \alpha) + \Pr(\hat{x} > \alpha, x < \alpha) + \Pr(\hat{x} < \alpha, x > \alpha)
\]
\[
= \varepsilon \left( \frac{\delta}{\beta} \right) + (1 - \varepsilon(\frac{\delta}{\beta}))[\Pr(\hat{x} > \alpha, x < \alpha) + \Pr(\hat{x} < \alpha, x > \alpha)] - \varepsilon(\frac{\delta}{\beta}) \Pr(\hat{x} < \alpha, x < \alpha)
\]
\[
\leq \varepsilon \left( \frac{\delta}{\beta} \right) + (1 - \varepsilon(\frac{\delta}{\beta}))[\Pr(\hat{x} > \alpha, x < \alpha) + \Pr(\hat{x} < \alpha, x > \alpha)]
\]

Moreover, we use the Total Probability Theorem once again and assume \( f(\xi) \) as the probability density function of \( x \):

\[
\Pr(\hat{x} > \alpha, x < \alpha) + \Pr(\hat{x} < \alpha, x > \alpha)
\]
\[
= \int_0^1 \Pr(\hat{x} > \alpha \mid x = \xi < \alpha) f(\xi) d\xi + \int_0^1 \Pr(\hat{x} < \alpha \mid x = \xi > \alpha) f(\xi) d\xi
\]
\[
= \int_0^\alpha \Pr(\hat{x} > \alpha \mid x = \xi) f(\xi) d\xi + \int_\alpha^1 \Pr(\hat{x} < \alpha \mid x = \xi) f(\xi) d\xi
\]

By noting that if \( \hat{x} > \alpha \) while \( x = \xi < \alpha \), it must have \( |\hat{x} - x| > \alpha - \xi \), we know that:
\[
\Pr(\hat{x} < \alpha \mid x = \xi > \alpha) \leq \Pr(|\hat{x} - x| > \alpha - \xi)
\]

Therefore

\[
\int_0^{\alpha} \Pr(\hat{x} > \alpha \mid x = \xi)f(\xi)d\xi + \int_{\alpha}^1 \Pr(\hat{x} < \alpha \mid x = \xi)f(\xi)d\xi \\
\leq \int_0^{\alpha} \Pr(|\hat{x} - x| > \alpha - \xi)f(\xi)d\xi + \int_{\alpha}^1 \Pr(|\hat{x} - x| > \xi - \alpha)f(\xi)d\xi \\
\leq \int_0^{\alpha} \varepsilon(\alpha - \xi)f(\xi)d\xi + \int_{\alpha}^1 \varepsilon(\xi - \alpha)f(\xi)d\xi \\
= \int_0^1 \varepsilon(|\alpha - \xi|)f(\xi)d\xi
\]

Therefore we finish the proof.

N.2 Proof of Theorem 2

Then for Theorem 2, consider the query \( \phi = \exists x_1, ..., x_n.\psi_1 \otimes \cdots \otimes \psi_m \), where \( \psi_i = (a_i, (\alpha_i, \beta_i)) \), the final error should be no more than a linear combination of each soft atomic query:

Proof.

\[
\begin{align*}
&\|\hat{U}[\phi](s) - U[\phi](s)\| \\
&= \max_{x_1=s_1, \ldots, x_n=s_n} (\hat{U}[\psi_1](s) + \cdots + \hat{U}[\psi_m](s)) - \max_{x_1=s_1, \ldots, x_n=s_n} (U[\psi_1](s) + \cdots + U[\psi_m](s)) \\
&\leq \max_{x_1=s_1, \ldots, x_n=s_n} \left( [\hat{U}[\psi_1](s) - U[\psi_1](s)] + \cdots + [\hat{U}[\psi_m](s) - U[\psi_m](s)] \right) \\
&\leq \max_{x_1=s_1, \ldots, x_n=s_n} \left( \| \hat{U}[\psi_1](s) - U[\psi_1](s) \| + \cdots + \| \hat{U}[\psi_m](s) - U[\psi_m](s) \| \right) \\
&\leq \sum_{i=1}^m \epsilon(\alpha_i, \beta_i)
\end{align*}
\]

The final line relies on the definition of \( \epsilon \), which gives the upper bound of error that only depends on \( \alpha, \beta \).

Reviewer paper summaries: This paper introduces a new task, complex soft query answering on uncertain knowledge graph. The authors define the task, proposed new method incorporating both forward inference and backward calibration. They also construct datasets for the complex soft query answering on UKG. The experiment results show that the proposed method SRC outperform the adapted classical query embedding methods. The comparison with LLMs is also conducted, showing that the complex soft query answering is not an easy task for LLMs and worth to investigate in the future.

Complex logical query answering are designed to answer queries over incomplete knowledge graphs. However, existing knowledge graphs have an inherent uncertainty, requiring new ML based query answering that can account for this uncertainty during the learning process. This paper introduces a novel framework for soft queries on uncertain KGs and proposes a neuro-symbolic method, called Soft Reasoning with Calibrated Confidence values (SRC), to answer such queries. The approach begins by introducing the syntax of soft queries (a disjunction of soft conjunctions) and then defines the semiring and a method for recursively calculating the confidence of a formula. A soft query graph is constructed, and a utility vector is computed to represent the distribution over all entities based on the formula. During forward inference, the edges of the soft query graph are progressively pruned to derive the final utility vector. Two strategies are proposed for score calibration: one involves debiasing, while the other is a learning approach with a trainable parameter. Alongside the proposed method, the paper provides theoretical analysis to ensure that the method avoids catastrophic error propagation. The proposed approach is compared with several baselines across three datasets—CN15k, PPI5k, and O*NET20k—using various metrics. The results demonstrate a significant improvement over the baselines.
Summary of strengths: 1. The new task complex soft query answering on UKG introduced in this paper is interesting and worth to investigate. It is a practical setting. 
2. The task definition, method, and dataset construction are introduced in a compact but clear way. I appreciate the authors tried to included so many new information and introduced them as detailed as possible in 9 pages. 
3. The Experiments show the proposed SRC method performs better than classical query embedding methods. More importantly, the classical query methods are directly applied to the new task but also been adapted, making them more strong baselines.

– The paper addresses a valid and important research gap in modeling uncertainty for complex query answering over uncertain KGs.

– The paper follows a logical progression in designing the model, and the method is technically sound.

– The results demonstrate a significant performance improvement over competitors across several datasets.
Summary of weaknesses: 1. The contents of the paper are very compact that the related work section is included in the appendix, which might cause understanding difficulties for readers that are not familiar with complex query answering tasks on KGs. 
2. Some of the explanations of symbols in the Equations are missing.

– The paper compares two old baselines with four variants of the proposed model. However, more recent SOTA, such as GNN-QE [1], are not included, which undermines the quality of the evaluation.

– The clarity of the paper needs further improvement. For instance, it is challenging to understand the details of the work and the sequence of steps from the high-level Algorithm 1, as well as from the description in Section 4.1. Additionally, a sentence like "We first remove the constant nodes by Lemma 1" lacks sufficient clarity, as Lemma 1 only indicates the existence of a transformation, making the explanation less informative.

–The validity of Assumption 1 in real-world KGs is unclear and requires further clarification, as it serves as the foundation for the main theorems.



[1] Zhu, Zhaocheng, et al. "Neural-symbolic models for logical queries on knowledge graphs." International conference on machine learning. PMLR, 2022.
Comments, suggestions and typos: ==Suggestions
1. The Equation (5) should be explained in details. 
2. Equation (9): the meaning of tho_theta and lambda_theta are not explained.
3. It would be better to include the statistics of the constructed datasets in the main contents.  

==Typos
1. Line 230: the disjunction of soft soft conjunctive query —> the disjunction of soft conjunctive query.

– please include the details of parameterization for equation 9 in the main body.
– page 3, definition 3, typo “soft soft”
– The model is not scalable as it requires to store vector with the size of number of entities, and also the complexity O(|R||E|^2) is very high. Line 433-437 in page 6 explain enhancing efficiency of the model with some techniques, but the description is very small and requires further explanation.
Overall assessment scores: 4

3