SConE: Simplified Cone Embeddings with Symbolic Operators for Complex Logical Queries
Year: 2023

Abstract:
Geometric representation of query embeddings (using points, particles, rectangles and cones) can effectively achieve the task of answering complex logical queries expressed in first-order logic (FOL) form over knowledge graphs, allowing intuitive encodings. However, current geometric-based methods depend on the neural approach to model FOL operators (conjunction, disjunction and negation), which are not easily explainable with considerable computation cost. We overcome this challenge by introducing a symbolic modeling approach for the FOL operators, emphasizing the direct calculation of the intersection between geometric shapes, particularly sector-cones in the embedding space, to model the conjunction operator. This approach reduces the computation cost as a non-neural approach is involved in the core logic operators. Moreover, we propose to accelerate the learning in the relation projection operator using the neural approach to emphasize the essential role of this operator in all query structures. Although empirical evidence for explainability is challenging, our approach demonstrates a significant improvement in answering complex logical queries (both non-negative and negative FOL forms) over previous geometric-based models.

heading: 1 Introduction
text: Answering complex logical queries is a fundamental task of knowledge graphs (KGs) (Bollacker et al., 2008; Vrandečić and Krötzsch, 2014; Speer et al., 2017; Fellbaum, 2010; Lehmann et al., 2015; Mitchell et al., 2018) for various purposes of individuals and businesses. Conventional methods, such as Hartig and Heese (2007); Schmidt et al. (2010), have been well-studied on complete KGs. However, these methods face challenges in incomplete and largely-scaled KGs, as conventional methods cannot traverse graphs via missing connections. Time complexity is another challenge as it grows exponentially during the traversal process. Modern
approaches, such as Hamilton et al. (2018); Ren et al. (2020), use query embeddings (QEs) methods that can answer complex logical queries without the need of path traversal in graphs. The QEs methods first transform a complex logical query into a machine-readable format: (1) converting a query in natural textual form into first-order logic (FOL) form (including conjunction ∧, disjunction ∨, negation ¬ and existential quantification ∃ operator) and (2) decomposing it into a computation graph (including relation projection operator). For example, Fig. 1 depicts the process of turning a complex logical query “Which universities do the Nobel Prize winners of Australian citizens work in?” into a computation graph. This FOL query is then projected in the embedding space, required for the modeling process to learn to answer the query.
Among different approaches in representing queries in the embedding space, geometric-based approaches have had renewed interests since the work in point embeddings (Hamilton et al., 2018). Following works have expanded this approach using hyper-boxes (Ren et al., 2020), sets of points as particles (Bai et al., 2022), hyperboloids (Choudhary et al., 2021b) and 2D-cones (Zhang et al., 2021). These works commonly resort to set operators over shapes that can handle the conjunction, only a few Zhang et al. (2021) can handle the negation. Nevertheless, existing geometric-based methods depend on the neural approach to model the conjunction operator. This approach is not easily explainable, counter-intuitive and does not take full advantage of the properties that these geometric representations are intended to be used for.
We highlight in this paper the essential role the projection operator plays in all complex query embedding methods. This operator is often learned through training neural architectures together with the logical operators in an end-to-end fashion. The semantic role of this operator is to obtain a meaningful representation of a predicate (relation) in
11931
FOL, which operates as a function converting a domain of input embeddings into a range of outputs. These are fundamentally different from the roles of the logical operators, in which geometric approaches can be modelled as set operators. Moreover, little work highlights the importance of the neural approach in learning the relation projection.
We address the first issue by introducing a novel symbolic operator in geometry for answering the conjunctive queries. Specifically, we directly calculate the intersection of geometric shapes in the embedding space (see an example in Fig. 2 and more details in Sec. 4.2). We use cone embeddings (Zhang et al., 2021) as a key geometric representation in our approach, since conic shapes were shown to be effective in modeling all FOL operators. By directly calculating intersection, our approach can reduce the computational cost of modeling the conjunction operator, as there is no need to incorporate expensive neural training in this logic operator, as compared with other geometric-based models (Hamilton et al., 2018; Ren et al., 2020; Choudhary et al., 2021b; Zhang et al., 2021). Further, classifying types of geometric intersection (partial, complete and none type) can improve the explainability in modeling conjunction operator (see Sec 4.2). To highlight the importance of relation projection (finding tail entities from a source entity via a relation) in complex logical query embeddings, we propose a general framework of modeling this operator, called relation projection network (RPN) (see Fig. 3). The RPN can enhance the learning in the relation projection operation, due to its high frequency and its dominance in diverse query structures (see Fig. 6).
Overall, we introduce Simplified Cone Embeddings (SConE) for modeling the relation projection and logical operators in complex queries. Our contributions are: (1) introducing a symbolic modeling for the conjunction operator in FOL query, (2) proposing a general framework using
RPN to improve the learning of relation projection operator in both atomic and complex logical queries and (3) surpassing model performance of previous state-of-the-art geometric-based models for both non-negation and negation queries.

heading: 2 Related Work
text: Atomic query answering for knowledge graph completion The atomic query has a given head concept (vh) and a relation (r), and the answering task is to find the projected tail concept (vt). Using geometric-based methods to answer atomic queries (or path queries) without complex logical operators has been well-studied since the appearance of knowledge graph embeddings, notably, translation-based methods (Bordes et al., 2013) and rotation (Sun et al., 2019; Zhang et al., 2020a). Further, Nickel and Kiela (2017); Balažević et al. (2019) proposed hyperbolic space (non-Euclidean geometry) over a Poincaré ball while others (Gao et al., 2020) used 3D shapes. However, these models are limited in answering complex queries involving FOL logical operators (e.g. Fig. 1 and Fig. 6).
Complex logical query answering for multi-hop reasoning The complex logical query has atomic queries with logical operators (see Fig. 1). Different methods addressing this task are geometrybased embeddings (points Hamilton et al. (2018), boxes Ren et al. (2020), hyperboloids Choudhary et al. (2021b), cones Zhang et al. (2021), particles Bai et al. (2022), distribution-based embeddings (Ren and Leskovec, 2020; Choudhary et al., 2021a; Huang et al., 2022; Yang et al., 2022; Long et al., 2022), auxiliary enrichment methods (Hu et al., 2022) using entity and relation type knowledge, logic-based methods (Arakelyan et al., 2021; Chen et al., 2022; Zhu et al., 2022; Xu et al., 2022) using fuzzy logic to model the logical operators, neural-based methods (Kotnis et al., 2021; Liu et al., 2022; Amayuelas et al., 2022) and oth-
ers (Sun et al., 2020). Although the logic-based methods are explainable in modeling FOL operators (a learning-free), other methods such as geometric-based embeddings are challenging to interpret this process, since these rely on the neural approach to model the conjunction operator. We provide a symbolic modeling approach for handling the conjunction to improve explainability in geometric-based models.

heading: 3 Preliminaries

heading: 3.1 First-Order Logic queries over KGs
text: Given a set of entities (v ∈ V) and a set of relations (r ∈ R), a knowledge graph (KG) G = {(vh, r, vt)} is a set of triples, each includes a head entity (vh), a relation (r) and a tail entity (vt).
Given a knowledge graph, a complex FOL query is a formula consisting of: constants, quantified bound variables (V1, . . . , Vn) and free variables (V?) (target), in addition to relation symbols R(Vi, Vj) and logic connectives (∃,∧,∨,¬). An entity of KG (v ∈ V) maps to each constant or variable. Each R(Vi, Vj) maps to a binary function whether a relation exists between (Vi) and (Vj). Logic connectives are conjunction (∧), disjunction (∨), negation (¬) and existential quantification (∃)1 (see an example of FOL query mapped to the (ip) structure in Fig. 1, and more query structures in Fig. 6). Given this example, the goal of a FOL query answering is to find the answers (or free variables) such that the formula is true.

heading: 3.2 Query Embeddings
text: Cone parameterization. We adopt the same definitions and propositions in Zhang et al. (2021), to define a two-dimensional sector-cone using two
1Universal quantification (∀) rarely appears in the real situation, this connective is therefore excluded.
variables: (1) angle α ∈ [−π, π) represents the angle between the semantic center axis and the positive x−axis, and (2) aperture β ∈ [0, 2π] represents the aperture of the sector cone (see an example with pink sector-cone in Fig. 2).
Query Embeddings representation. Given a complex logical query (q), we represent its embedding (q) as a Cartesian product of two-dimensional sector-cones in the embedding space using two variables: semantic center axis αq ∈ [−π, π)d and aperture βq ∈ [0, 2π]d, where d is the embedding dimension. Next, given a semantic entity (v), we represent its embedding (v) as a Cartesian product of cones embedding using semantic center axis αv ∈ [−π, π)d and zero aperture defined by:
q = (αq,βq), v = (αv,0) (3.1)

heading: 3.3 Operators in First-Order Logic queries
text: We decompose the symbolic representation of the complex logical query (q) using a computation graph, a tree-like query (see Fig. 1). This graph has vertexes and links where each vertex represents a set of entities and each link represents a modeling process of either of two types: relation projection operator (P) or any FOL operators (conjunction (C), disjunction (D) and negation (N )):
• Relation Projection (Projection): P(x, r) computes the projection from the input (x) as a head entity to the set of tail entities via relation (r). Otherwise, P(x, r−1) computes the projection from the input (x) as a tail entity to the set of head entities via (r).
• Conjunction (Intersection): C(x1,x2) computes the intersection of each geometric element in one set of entities (x1) and the corresponding element in the other entity set (x2).
• Disjunction (Union): D(x1,x2) computes the union of each geometric element in one set of entities (x1) and the corresponding geometric element in the other entity set (x2).
• Negation (Complement): N (x) computes the complement of each geometric element in the set of entities (x).

heading: 4 Modeling operators of FOL queries
text: We describe the modeling of relation projection (4.1) and logical operators (4.2) in a complex FOL query (q) (with its set of answer entities Vq ⊂ V) over knowledge graphs in the following:

heading: 4.1 Modeling relation projection
text: This section is to model the relation projection operator (P) (see Sec. 3.3) for Knowledge Graph Embedding (KGE). Overall, given an atomic query q = (v, r), we propose a relation projection network (RPN) with two layers: (1) first transforming the source entity using an ensemble of multiple KGE techniques, (2) merging the outputs at the second layer (called entanglement layer) to produce the sector-cones embedding of the query (see Fig. 3). We use two KGE techniques at the first layer, called relation transformation and multilayer perceptron. As the relation projection task is similar to KGE, one can select different models in KGE such as TransE (Bordes et al., 2013) or HAKE (Zhang et al., 2020b), then adapt these into the first layer in principle.
There are many models achieving KGE (Ji et al., 2022), the ensemble selection is therefore not restricted, but it should be efficient in model complexity and computational cost. Fundamentally, using one technique is sufficient; however, having a general framework using multiple techniques is to analyze the learning process from a broader viewpoint. We select two KGE models as a simple case
to illustrate that it is possible to use multiple KGE techniques.
Relation transformation Specifically, we model an embedded relation r = (Wr,br) requiring for the projection operation (P) by a neural network as in (Chen et al., 2022), where Wr denotes a weight matrix and (br) denotes a bias vector. We transform a source entity v = (αv,βv) into an embedded query (q) via this relation. However, as our entity representation based on sector-cones embeddings, which is different than the fuzzy sets used in (Chen et al., 2022), we add a concatenation operation of the semantic center axis (αv) and the aperture (βv) to convert these into a vector [v] ∈ R2d as follows:
qt = f(v) = LN(Wr[v] + br),
where LN is Layer Normalization (Ba et al., 2016). We use the basic decomposition of (Schlichtkrull et al., 2018) to define (Wr) and (br).
Multi-layer Perceptron (MLP) An alternative way to model the relation projection is to use MLP. We transform the entity (v) to query (q) via the relation (r) by a mapping function (f) as follows:
qm = f(vr) = LN(MLP([vr])),
where MLP : R2d → R2d is to approximately represent the mapping function f(x), vr is a translation embeddings of the source entity and the relation: vr = v+r. As the representation of the entity and the relation r = (αr,βr) are sector-cones embedding, we apply a concatenation operation as that in the relation transformation technique to convert (vr) to the vector embedding [vr] ∈ R2d. Entanglement layer After transforming the entity to the embedded query using the relation transformation and the MLP, we introduce an entanglement layer to merge the output from the first KGE layer into one output. We use attention mechanism in this layer:
q = (αq,βq) = s (∑2 iA [qt,qm] ) ,
where s(x) is a function to split the 2d-vector into two d-vectors, each is for the semantic center axis and the aperture embedding of the query, denotes Hadamard product, [, ] denotes an operator to stack two 2d-vectors into a matrix in R2×2d and A ∈ R2×2d is an attention matrix as follows:
A = SoftMax (fa(qt,qm)) ,
where the SoftMax(.) function applies over the first dimension of the matrix. The fa(qt,qm) = MLP([qt,qm]) is attention score function. We also provide a scaling function to convert the semantic center axis (αq) and the aperture (βq) into their normal range (see Appendix B.1) as that in ConE (Zhang et al., 2021).

heading: 4.2 Symbolic modeling of logical operators
text: In this section, we describe the modeling process of all logical operators (C,N ,D) using symbolic modeling only without neural-based methods, naturally making use of the geometric properties of sector-cone shapes in the embedding space. In comparison with ConE (Zhang et al., 2021), this model leveraged the neural approach to learn the conjunction (C) while using non-neural approach to model the disjunction and negation (D,N ).
Conjunction This aims to model the conjunction C(q1, q2) of any pair of conjunctive queries, each query qi = (αi, βi) is in the cone embedding space. Assuming the embedding dimension (d = 1) as the simplest case, each query is represented by a sectorcone (see Eq. (3.1)). As the intersection of the two conjunctive queries is also a sector-cone q∧ = (α∧, β∧); therefore, one can directly calculate this intersection from a symbolic geometric perspective as follows:
β∧ =    u2 − l1, β2, 0, u1 − l2, β1,
0,
α∧ =    u2 − β∧2 , if c1 α2, if c2 l1 − |l1−u2|2 , if c3 u1 − β∧2 , if c4 α1, if c5 l2 − |l2−u1|2 , if c6
(4.1)
where (ui, li) is the upper and lower bound for each sector-cone (li ≤ αi ≤ ui). These calculations are that (ui = αi + βi2 ) and (li = αi − βi 2 ); and ci represents each conditional scenario regarding relative position between the two sector-cones:
c1 := (u1 ≥ u2) ∧ (u2 ≥ l1) ∧ (l1 ≥ l2), c2 := (u1 ≥ u2) ∧ (u2 ≥ l2) ∧ (l2 > l1), c3 := (u1 ≥ l1) ∧ (l1 > u2) ∧ (u2 ≥ l2), (4.2) c4 := (u2 ≥ u1) ∧ (u1 ≥ l2) ∧ (l2 ≥ l1), c5 := (u2 ≥ u1) ∧ (u1 ≥ l1) ∧ (l1 > l2), c6 := (u1 ≥ l2) ∧ (l2 > u1) ∧ (u1 ≥ l1).
Note that there are three types regarding calculating intersection of two sector-cones in Eq. (4.2): (1) partial intersection (see c1, c4), (2) complete intersection (see c2, c5) and (3) none intersection (see c3, c6). Figure 4 shows these cases (c1, c2, c3)
𝑞2𝑞1 ∧ 𝑞2𝑞1 𝑞2𝑞1 ∧ 𝑞2 𝑞1 ∧ 𝑞2𝑞2𝑞1(𝑐1) Partial (𝑐2) Complete (𝑐3) None 𝑢1 𝑢2 𝑙1 𝑢1 𝑢2 𝑢1 𝑙1 𝑢2 𝑙2
from one sector-cone to the other and vice versa (c4, c5, c6). While the calculation of the partial and complete intersection are based on natural representation of geometric shapes, the calculation of none intersection type is based on zero aperture and middle semantic axis between the lower bound of one sector-cone and the upper bound of the other. The aperture in this situation is confidence, but the semantic axis is uncertain as it can be any axes between the mentioned bounds. We consider the middle axis as a special case for none intersection type (see further details in the following Eq. 4.3).
In general, to compute the intersection of (k) conjunctive queries, assuming this computation satisfies the associative and/or commutative law for logic, we compute the intersection C(qi, qi+1) of the first two arbitrary conjunctive queries, then compute the intersection of C(qi, qi+1) and the next conjunctive query (qi+2) to produce C(C(qi, qi+1), qi+2), and iterate this process until reaching the final conjunctive query (qk).
𝑞
Conjunction: Weight semantic axis for none type intersection In the conjunction C(q1, q2) with none type intersection (c3, c6), the equality of axis intersection (α∧) of two sector-cones embeddings can be any axes between the upper bound of one sector-cone and the lower bound of the other sector-cone. In general, the equality to calculate (α∧) in the cone embedding space (see Eq. 4.1) for
the none type intersection is shown below:
α∧ = { δl1 + (1− δ)u2, if c3, δl2 + (1− δ)u1, if c6,
(4.3)
where (δ ∈ [0, 1]) is a hyper-parameter to control the spatial location of axis intersection regarding that of the two mentioned bounds. Notice that, let (δ = 0.5) as in Eq. (4.1), the semantic center axis intersection (α∧) is in the middle between the two bounds as a special case of equality in Eq. (4.3) (see Appendix F for further analysis).
Negation This aims to model the negationN (q), called q¬ = (α¬,β¬) of the embedded query q = (α,β). In the cone embedding space, the semantic center axis of (q¬) should be in opposite direction via the O−axis regarding that of the (q) (see Fig. 5 (c)). In terms of the aperture, the summation of both apertures of (q¬) and (q) should be close to (2π) as follows:
α¬ = { α− π, if (α ≥ 0) α+ π, if (α < 0) β¬ = 2π − β.
Disjunction Similar to cone embedding Zhang et al. (2021), we adapt the DNF technique in Ren et al. (2020) to represent the disjunction operation D(q1, q2) as disjunction of conjunctive queries (see Fig 5 (b)). Hence, we can leverage (C,N ) operators above to have a set of embeddings of the conjunctive queries. Those entities nearest to any of these conjunctive queries in the cone embedding space are considered to be the answers (see the aggregated distance score in Eq. (4.5)).

heading: 4.3 Optimization
text: Distance score function We define a distance score function d(v,q) of the embedding between: the expected entity v = (α,0) and the query q = (αq,βq) (as stated in Sec. 3.2). We use two distance types: (dcon) for conjunctive queries and (ddis) for disjunctive queries as (Ren et al., 2020; Zhang et al., 2021). In (dcon), there are three terms: an outside distance (do), an inside distance (di) and a separated axis distance (da) (see Fig. 5 (d)), which are defined by:
dcon(v,q) = (1− ψ)(do + λdi) + ψda, (4.4)
where λ ∈ (0, 1) is to encourage (v) to be covered by the sector-cone embedding (q). The hyperparameter (ψ) is to weight the effect of the outside with inside distance and the separated axis
distance (see Appendix C for more details). To calculate (ddis), we use the DNF technique in Ren et al. (2020), which obtains the minimum distance in embeddings between: an expected entity and each conjunctive query in DNF, over a (k) number of conjunctive queries:
ddis(v,q) = min{dcon(v,qi)i:1→k}, (4.5)
Loss function During the optimization process, we use the negative sampling loss (L) (Mikolov et al., 2013a,b) as that in Ren and Leskovec (2020): L = L1 + L2, where L1 = − log σ(γ − d(v,q)) involves a minimization of the distance d(v,q) for a positive answer entity (v ∈ Vq), and L2 = − 1n ∑n i log σ(d(v ′,q) − γ) involves a maximization of the distance d(v′,q) for a number (n) of negative answer entities (v′i:1→n /∈ Vq); σ(x) is the activation function (e.g. sigmoid) and (γ) is a positive margin as hyper-parameter.

heading: 5 Experiments

heading: 5.1 Experimental setups
text: Multi-hop Reasoning (MHR) or Complex Logical Query Answering task Given an arbitrary complex FOL query, when traversing the incomplete KGs, non-trivial answers cannot be returned directly. The MHR task aims to find these answers. We evaluate our approach on there datasets: FB15k (Bollacker et al., 2008), FB15k-237 (Toutanova and Chen, 2015) and NELL995 (Xiong et al., 2017), following the preprocessing in BetaE (Ren and Leskovec, 2020). We follow the training protocol of previous works (Ren and Leskovec, 2020; Zhang et al., 2021), using 10 query syntaxes (non-negation 1p/2p/3p/2i/3i and negation 2in/3in/inp/pni/pin) for the training. We use these 10 syntaxes plus 4 unseen syntaxes (ip/up/2u/pi) for the evaluating process (see Fig. 6). An example of the (1p) query is (v, r1) i.e. (Wesleyan_University, major_field_of_study), while (2p) or (3p) query corresponds to (v, r1, r2) or (v, r1, r2, r3).
Evaluation Protocol Following the evaluation protocol in (Ren et al., 2020), given a query, we split its answers into two sets: easy answers and hard answers. The former is for those entities that can be reached on the training/validation graph through symbolic approach in graph traversing. The latter is for those that can be predicted using query embedding models, or the reasoning process
performs on hard answers. We use the mean reciprocal rank (MRR) metrics, computing the ranking of each hard answer against all non-answer entities, to measure the performance of models.
Baselines We use four recent geometric-based embedding models as baselines: GQE (Hamilton et al., 2018), Query2Box (Q2B) (Ren et al., 2020), Query2Particles (Q2P) (Bai et al., 2022) and ConE (Zhang et al., 2021), and obtain their results from ConE and Q2P. We also compare these results with state-of-the-art models based on fuzzy logic (see Appendix E.2).

heading: 5.2 Results
text: Existential Positive First-order (EPFO) queries Overall, the average MRR in all EPFO queries without negation (AVGp) of SConE2 significantly outperform all geometric-based baselines using the three datasets, particularly more than that of ConE by nearly 12% using the NELL995 dataset (see Table 1). For queries (1p/2p/3p/2i/3i) involving in
2Source code is available at https://github.com/nlptlp/scone
the training process, most of the average MRR in each of these query structure (11 out of 15 metrics) significantly surpass baselines. Specially, in the (2p) query, around 26% gain of the MRR in SConE over that in ConE observes in the NELL995 dataset. With regard to queries (ip/pi/2u/up) that are not involved in the training process, the model performance of SConE also shows a significant increase of MRR, compared to that of ConE (10 out of 12 metrics), which suggests an improvement in the ability of zero-shot learning for these queries (please see Appendix E.1 for error bars of the main results).
Negation queries Overall, the average MRR in negation queries (AVGn) of SConE is significantly higher than that of ConE by closely 14% using the FB15k-237 dataset (see Table 1); even though there is no difference in the modeling of negation operator in both models. This can be due to the effect of using RPN to enrich learning in the atomic query structure (1p), which dominantly involves in all negation queries (see Fig. 6 Bottom-Left and Sec. 5.3 for further ablation study in the RPN).

heading: 5.3 Ablation study - Sensitivity analysis
text: We conduct experiments for ablation study w.r.t. two situations: relation projection networks and geometric intersection types; and for sensitivity analysis w.r.t. two hyper-parameters: distance weight (ψ) and embedding dimensions (d) as follows:
Relation projection network Table 2 shows the average MRR on FOL queries in general and several specific EPFO queries (e.g. 1p/2i/ip/2u) of the test set w.r.t. different RPNs: (1) MLP only, (2) relation transformation only and (3) MLP with relation transformation and attention mechanism for the entanglement layer (see Sec. 4.1). Overall, the third scenario observes the highest model’s performance of answering complex logical queries over the other scenarios. We attribute this observation to the advantage of enhancing the learning for atomic query (1p) using the RPN, resulting in an improvement in model performance in total. Specifically, an increase from 42.5 (using MLP only) to 44.2 (using both MLP and relation transformation) in the average MRR (%) of 1p query can lead to an increase in that of other query structures (2i, ip, 2u). This is because the atomic query involves in all structures and in the early stage during the decomposition process via the computation graph for each structure (see Fig. 6). Moreover, SConE with MLP only (d = 800, around 20.0M parameters) and symbolic modeling for logical operators uses less parameters than those in ConE (d = 800, around 23.9M parameters reported in Long et al. (2022) Appendix B), but both models achieve similar performance (see Table 1).
Geometric intersection of sector-cones Table 3 shows the model’s performance of answering complex logical queries using different types of intersection between sector-cones. The implementation of using individual intersection type is to see the impact of each intersection type on model performance, compared with that using all conditional
intersection types. Geometrically, there are three types (None, Complete, Partial) of intersection for any pair of two conjunctive queries as illustrated in Figure 4. Using one type of intersection calculation individually will result in misrepresentation for the other two. For example, assuming all query pairs have None intersection, i.e. only using (c3) and (c6) (see Eq. 4.2) for intersection calculation, we will miss the opportunity to capture Complete and Partial overlap correctly. Table 3 confirms our intuition, the model performs best when all intersection types are considered. Notice that the calculation of intersection of sector-cones neglects the neuralbased approach, but the model is able to efficiently learn to answer intersection queries as that in ConE, particularly when using partial intersection only. It is arguable that the learning process focuses on the atomic query which are also involved in conjunctive queries.
Weight distance Table 4 (Left) shows the average MRR on FOL queries of the test set w.r.t. different weights (ψ) of distances (see Eq. (4.4)), ranging from zero (no axis distance but having inside and outside distance), half of one (equally having axis distance with inside and outside distance) to one (having axis distance but no inside and no outside distance). The model performance increases from (ψ = 0) to (ψ = 1), which suggests that there is an effect of using the axis distance (da). The performance reaches its peak when (ψ) is set to one as maximum, which suggests that the model can learn to answer complex logical queries using the axis distance only (without inside and outside distance). However, we theorize that the inside and outside distance should be involved during the training process. This is to improve the explainability of cone embeddings, where those entities inside the sector-cone are expected to be answers of the query. In this situation, the aperture plays a role in covering answer entities. Thus, to keep all distance types during the optimization process,
we set 0 < ψ = 0.9 < 1 for the main results (see Table 1).
Embedding dimension Table 4 (Right) shows the average MRR on FOL queries of the test set w.r.t. different embedding dimension. The model’s performance increases from using small (d = 64) to medium (d = 512) embedding dimension. This observation suggests that there is a significant effect of this hyper-parameter on the average MRR (of both EPFO and negation queries). Additionally, in the case d = 256 (with around 16.3M parameters), SConE uses less 30% in the number of parameters than those in ConE (d = 800 with around 23.9M parameters), but both achieves similar average MRR using the same FB15k-237 dataset.

heading: 6 Conclusions
text: We have provided a symbolic modeling for logical operators, particularly computing geometric intersection of sector-cones for modeling the conjunction. In addition, we highlighted the importance of the projection operator by introducing a relation projection network using neural-based approach, to strengthen the learning in atomic queries involved in all FOL query syntaxes. Our neural-symbolic approach using geometric embeddings significantly outperforms state-of-the-art geometric-based models in both EPFO and negation queries.
Limitations
Although our geometric embedding approach can handle a complete set of basic FOL operators (existential quantification, conjunction, disjunction and negation), the modeling of negation operator cannot narrow down the predicted answers to relevant topics of atomic queries. For example, one can expect the answers of this negation question/query “List Argentina players who are not Lionel Messi in World Cup 2022?” to be any teammates of Lionel Messi (i.e. 2in query structure). However, the
current model is designed to return all elements in the entire entity set except for Lionel Messi, which have redundant objects (e.g. trees, music, houses). This is a common limitation not only in geometric-based models but in others using fuzzy sets representation. This is due to the fact that the modeling of negation operator is assumed to be the complement set of a questionable entity w.r.t. the entire entity set. Our hypothesis is that the expected answers should be narrowed into the complement set w.r.t. a sub-topic of relevant entity set.
In addition, when apertures of two sector-cones are obtuse angles, the current calculation of partial intersection cannot correctly model the conjunction operator. This special case is inevitable in a system using geometric representation that is closed under negation and conjunction, but not for disjunction (see Appendix A.2 for further details).
Ethics Statement
The ability of models to answer complex logical queries is achievable to reason about knowledge graphs. Due to model’s uncertainty, one potential negative impact of this task is the out-of-control in automatic reasoning over open large-scale knowledge graphs, where there are diverse source of information. Some of which though can be missed from KGs due to incompleteness or private purposes, but they can be possibly reasoned using the query embedding methods.

heading: Acknowledgements
text: This research is supported by the Australian Research Council through the Centre for Transforming Maintenance through Data Science (grant number IC180100030), funded by the Australian Government. Wei Liu acknowledges the support from ARC Discovery Projects DP150102405. Further, the authors would like to thank all the anonymous reviewers for their insightful feedback.

heading: A Modeling logical operators - None intersection type
text: A.1 Comments on partial and complete type of intersection
Note that the condition (c1) for partial intersection type has a special case when these equalities holds (u2 = l1) and (l1 = l2). In this situation, the partial type (c1) can be considered as the complete type (c2). Hence, these types of intersection can be used interchangeably. Similarly, we can interchangeably use the calculation of the partial and complete type for the condition (c4, c5). This is due to the fact that the calculation of these intersection types become as follows:
β∧ =    0, β2, 0,
β1
α∧ =    u2, if c1, α2, if c2, u1, if c4, α1, if c5.
In terms of the cases (c1, c2), notice that (u2 = l1 = l2) or (u2 = l2), hence (0 = β2), and (u2 = α2 + β2 2 ) or (u2 = α2). Similar explanation is for the cases (c4, c5).
A.2 Special case of partial intersection In case of the aperture of two sector-cones are both obtuse angles as shown in Fig. 7, the calculation of partial intersection under conditions (c1, c4) cannot exactly model the conjunction operator. Specially, the calculation of (α∧, β∧) for conjunction operator in Sec. 4.2 will return (α∧, β∧) of the right intersection sector-cone, but will ignore (α∧, β∧) of the left intersection sector-cone in this special case. Our hypothesis is that this special case is in-
evitable if a system using geometric representation, such as conic shapes, is closed under negation and conjunction, but not disjunction. This limitation can be addressed by providing a refined approach for calculating the partial intersection when both apertures are obtuse angles. We leave this approach as an extension for future work.

heading: B Further details in section modeling operators of FOL queries
text: B.1 Scaling function Continuing Sec. 4.1, after obtaining the output from entanglement layer as q = (α,β), we scale the semantic center axis and the aperture into their normal ranges [−π, π) and [0, 2π] respectively, defined in Sec. 3.2. The final embedded query q = (α′,β′) is as follows:
α′ = π tanh(λ1α), (B.1) β′ = π tanh(λ2β) + π, (B.2)
where (λ1, λ2) are scaling hyper-parameters.

heading: C Distance score functions
text: Continuing the calculation of distance score function in Sec. 4.3, the calculations of the outside with inside distances and axis distance are as follows:
do = ∣∣∣ ∣∣∣min{dl, du} ∣∣∣ ∣∣∣ 1 , di = ∣∣∣ ∣∣∣min{dα, dβ} ∣∣∣ ∣∣∣ 1 ,
da = ∣∣∣ ∣∣∣α−αq ∣∣∣ ∣∣∣ 1 , (C.1) where || · ||1 denotes the L1 norm, the upper bound (u = αq+ βq 2 ) and the lower bound (l = αq− βq 2 ) are of the query (q); dl = |1 − cos(α − l)| and du = |1−cos(α−u)| is the lower and upper bound outside distance respectively, dα = |1− cos(α− αq)| and dβ = |1 − cos(βq2 )| is the axis and the aperture inside distance respectively.

heading: D Experimental setups
text: D.1 Datasets
Following experimental settings in Ren and Leskovec (2020) for the training and evaluation process, which pre-processed datasets (FB15k Bollacker et al. (2008), FB15k-237 Toutanova and Chen (2015) and NELL995 Xiong et al. (2017)) and publicly available at this link 3 , Table 5 shows statistics of these datasets regarding the number of entities, the number of relations and the number of triples. In addition, Table 6 shows the number of queries in different structures for the training/validation/test set.
D.2 Training and evaluation settings: Further details, hyper-parameters and error bars
Following the original work in (Ren and Leskovec, 2020) 4, we implement all experiments using Pytorch as Deep Learning framework under Python. For each experiment, we conduct it on a single NVIDIA Tesla V100 GPU, in the UWA Kaya High Performance Computing (HPC) cluster. For the settings in the relation projection network, we use a three-layer MLP using 1600 dimension for hidden layers and Rectified Linear Units (ReLU) for the activation function. Further, following the found hyper-parameters in ConE (Zhang et al., 2021): λ1 = 1.0, λ2 = 2.0, λ = 0.02, batch size b = 512 and negative sampling size n = 128, we use these in all experiments. With regard to other hyperparameters, we search for the best performance
3https://github.com/snap-stanford/KGReasoning 4https://github.com/snap-stanford/KGReasoning,licensed
under the MIT License
in MRR. Specifically, (γ) involving in the loss function is in {20, 30}, the learning rate (l) is in {1e−4, 5e−5}. Table 7 shows found hyperparameters of the main results in Table 1. For obtaining error bars of the main results, we run the model five times, each uses different random seed in {0, 10, 100, 1000, 10000} (see Table 8 for further details).

heading: E Additional results
text: E.1 Error bars for the main results
Table 8 shows error bars of the average MRR (in percentage) for the main results of SConE reported in Table 1 (see random seed settings as described in Appendix. D.2). We compute the standard deviation (std) of results from five experiments using each of the three dataset (FB15k, FB15k-237 and NELL995). Overall, the error bar of the average MRR is low in all query structures and in average for EPFO and negation queries, which demonstrates the stability of model performance.
E.2 Comparison results with fuzzy logic-based models
Table 9 shows comparison in the average MRR (in percentage) of SConE with that of other fuzzy logic-based models. In the FB15k-237 dataset, GNN-QE achieves state-of-the-art performance of answering complex logical queries. Compared to other models (ENeSy and FuzzQE), the performance of SConE is nearly to that of these logicbased models. With regard to the NELL995 dataset, although SConE achieves the lowest performance in answering negation queries, the performance of SConE reaches its highest in answering non-
negation queries among other logic-based models. Particularly, there is a significant improvement in the average MRR regarding union queries, compared to that in other models.

heading: F Further sensitivity analysis - Weight of semantic axis for none intersection type
text: Table 10 shows the performance of SConE w.r.t. different weights (δ ∈ [0, 1]) of semantic axis in the case of none type intersection (see Eq. (4.3)). We conduct five experiments using different weights in {0.0, 0.1, 0.5, 0.9, 1.0}. When (δ = 0.0) or (δ = 1.0), the semantic axis of an intersection query corresponds to the lower bound of one sectorcone or the upper bound of the other sector-cone. When (δ = 0.1) or (δ = 0.9), the spatial position of this semantic axis is close the lower bound of one sector-cone or the upper bound of the other sector-cone respectively. In a special case when (δ = 0.5), the semantic axis is in the middle of the two bounds.
Overall, the average MRR (AVG) of SConE for both non-negation and negation queries is similar from one to another in all different weights (δ). However, there is a slight difference between AVGp
for non-negation queries and AVGn for negation queries. When (δ = 0.1), SConE achieves the highest AVGp but not for AVGn. In contrast, when (δ = 0.5), SConE achieves the highest AVGn but not for AVGp. Since there is a slight difference in AVGp using (δ = 0.1) and (δ = 0.5) but there is highly difference in AVGn using these weights, we select the special case with (δ = 0.5) or the middle semantic axis of intersection query and report the main results. Further, we observe that there is no significant difference in the average MRR (AVG) of model performance for both non-negation and negation queries (see second column of Table 10). Thus, any semantic axes (or cones) between the two mentioned bounds can be considered as the semantic axis of intersection query. Note that the aperture of intersection query in the case of none intersection type is equivalent to zero.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section Limitations (after Section 6 Conclusion), Appendix A.3
3 A2. Did you discuss any potential risks of your work? Section Ethics Statement (after Section Limitations)
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5.3, Appendix D.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5.1, Appendix D.1, D.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix D.2, Appendix E.1
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Appendix D.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave this approach as an extension for future work