SlowGuess/NAACL_2022_OCR huggingface dataset:
do not appear in the training set and are directly evaluated in a zero-shot way. In another work, Ren and Leskovec (2020) refine these queries by raising the difficulties of the existing nine types of queries. They also include five types of complement queries (2in, 3in, inp, pni, pin) for general first-order logic (FOL) queries. These complement queries are also trained and evaluated in the supervised setting, but their training samples are fewer than other types. More details about the knowledge graphs and sampled queries are shown in the appendix. To demonstrate the performance of Query2Particles, it is evaluated on both sets of queries. Note that, the query-answer pairs used for training are only from the training graph  $G_{train}$ . For validation and testing, only the hard answers from validation graph  $G_{valid}$  and testing graph  $G_{test}$  are evaluated.
The Query2Particles model is compared with the following baselines in the following sections.
Graph Query Embedding (GQE) answers conjunctive logic queries by encoding the logical queries into vectors (Hamilton et al., 2018).
Query2Box (Q2B) answers existential positive first-order logic queries by encoding them into boxes in the embedding space (Ren et al., 2020).
Beta Embedding (BetaE) answers first-order logic queries by modeling them as Beta Distributions (Ren and Leskovec, 2020). This is the current state-of-the-art model on first-order logic queries.
The reported mean reciprocal rank (MRR) scores of these baselines are used by the BetaE paper (Ren and Leskovec, 2020), and Query2Particles (Q2P) is evaluated following with the same metrics under the filtered setting, in which the rankings of answers are computed excluding all other correct answers. Meanwhile, the Q2P method is also compared with other methods on EPFO queries with the queries used by Ren et al. (2020).
Continuous Query Decomposition (CQD) decomposes the complex queries to multiple atomic queries that can be solved by link predictors (Arakelyan et al., 2020).
**Embedding Query Language (EmQL)** improves the faithfulness in the reasoning process by encoding EPFO queries into centroid-sketch representations (Sun et al., 2020).
The reported Hit@3 results of these two baselines are used by Arakelyan et al. (2020); Sun et al. (2020). Our model is evaluated on FB15K, FB15K-237, and NELL in the same setting.
The Query2Particles model is trained on the queries in an end-to-end manner. To fairly compare with previous methods, we set the same size of embedding vectors as four hundred. We use the validation queries to tune hyperparameters for our model by using grid search. In the grid search, we consider the batch size from  $\{1024, 2048, 4096, 8192\}$ , dropout rate from  $\{0.1, 0.2, 0.3\}$ , learning rate
DATASET	MODEL	2IN	3IN	INP	PIN	PNI	AVG
MRR	H@10	MRR	H@10	MRR	H@10	MRR	H@10	MRR	H@10	MRR	H@10
FB15K	BETAE	14.3	30.8	14.7	31.9	11.5	23.4	6.5	14.3	12.4	26.3	11.8	25.3
Q2P (OURS)	21.9	41.3	20.8	40.2	12.5	24.2	8.9	18.8	17.1	33.6	16.4	31.6
FB15K-237	BETAE	5.1	11.3	7.9	17.3	7.4	16.0	3.6	8.1	3.4	7.0	5.4	11.9
Q2P (OURS)	4.4	10.1	9.7	20.7	7.5	16.7	4.6	9.9	3.8	7.2	6.0	12.9
NELL	BETAE	5.1	11.6	7.8	18.2	10.0	20.8	3.1	6.9	3.5	7.2	5.9	12.9
Q2P (OURS)	5.1	12.1	7.4	18.2	10.2	21.4	3.3	7.0	3.4	7.6	6.0	13.3
AVERAGE	BETAE	8.2	17.9	10.1	22.5	9.6	20.1	4.4	9.8	6.4	13.5	7.8	16.7
Q2P (OURS)	10.5	21.2	12.6	26.4	10.1	20.8	5.6	11.9	8.1	16.1	9.4	19.3
Table 2: The MRR and Hit@10 results for complement queries used by Ren and Leskovec (2020). Beta Embedding is the only baseline model that can answer queries with the complement operation.
FB15K	FB15K-237	NELL
MODEL	1P	2P	3P	2I	3I	IP	PI	2U	UP	AVG	Avg	AVG
EMQL	42.4	50.2	45.9	63.7	70.0	60.7	61.4	9.0	42.6	49.5	35.8	46.8
-SKETCH	50.6	46.7	41.6	61.8	67.3	54.2	53.5	21.6	40.0	48.6	35.5	46.8
CQD-BEAM	91.8	77.9	57.7	79.6	83.7	37.5	65.8	83.9	34.5	68.0	29.0	37.6
CQD-CO	91.8	45.4	19.1	79.6	83.7	33.6	51.3	81.6	31.9	57.6	27.2	36.8
Q2P (OURS)	90.2	74.6	73.4	86.0	89.6	63.7	77.6	83.4	52.7	76.8	43.0	52.2
Table 3: The Hit@3 results for existential positive first-order queries originally used by Ren et al. (2020) and the comparisons are made against the state-of-the-art baselines including EmQL and CQD methods. The best results are marked in bold and the second-best ones are marked with underlines. The full results are in the appendix.
from  $\{10^{-4}, 3 * 10^{-4}, 10^{-3}\}$ , and label smoothing from  $\{0.3, 0.5, 0.7\}$ . The final hyperparameters are shown in the supplementary materials. Our experiments are conducted on Titan Xp with PyTorch 1.8, and they are repeated three times.
5.4 Comparison with Baselines
First, we compare Query2Particles (Q2P) with GQE, Q2B, and BetaE on the first-order logic queries used by Ren and Leskovec (2020). The results on all fourteen types of queries are reported in Table 1 and Table 2. To fairly compare with the baseline methods, we keep the same number of parameters used in each type of query embedding.
As shown in Tables 1 and 2, the Q2P model can achieve more accurate results than GQE, Q2B, and BetaE on all types of queries except  $2\mathbf{u}$ . As we keep the number of query embedding parameters the same, it indicates that the structure of particle embeddings is more suitable for encoding complex queries than boxes or Beta distributions.
Though it is slightly less accurate on the  $2\mathbf{u}$  queries, Q2P is more efficient in encoding the queries that include union operations. This is because Q2P is the first embedding method that directly models the union operation. To avoid direct modeling of the union operation, all previous embedding methods pre-process the queries by con
verting them to DNF forms. However, the DNF forms can be exponentially larger than the original queries, and the conversion also takes exponential time. Meanwhile, BetaE proposes to use De Morgan's law to replace one union operation with one intersection and three complements, but this substitution still largely increases the query complexity. Instead, Q2P directly models the union operation without any pre-processing or additional parameterization, while achieving the state-of-the-art performance on up, which is more complicated and involving the Union operation.
We also compare our model with EmQL and CQD methods on the queries used by Ren et al. (2020). On average, our model has better Hit@3 scores on all datasets<sup>3</sup>. Compared to the CQD method, the Q2P method is better at answering multi-hop queries. encodes the complex queries into centroid-sketch representations, which cannot compactly encode sufficiently diverse answers. The Q2P method specifically addresses the diversity of answers, so it has higher empirical performance. CQD performs better on shorter queries like 1p, 2p, and 2u, because it can use the state-of-the-art link predictors. Also, as shown in Figure 6, the Q2P method demonstrates a faster inference
3In this paper, we only focus on the inductive setting, so we skip the comparison with EmQL under the entailment setting, in which the test graph is used for both training and testing.
Figure 6: The inference time of different types of queries on FB15k, which has the largest number of edges among three graphs.
speed than the CQD method on multi-hop queries, because CQD uses inference time optimization, which is either a continuous optimization or a beam search. The inference time optimization simplifies the learning of CQD but also slows down the inference efficiency on large graphs.
5.5 The Improvement of Q2P-KP
Experiments show that the performance of the diversified queries can be largely improved by using more particles. To demonstrate the effects, we conduct additional evaluations on the most diversified  $10\%$  queries for each query type, as shown in the DIVR columns in Table 4. In doing so, we use the number of answers to measure the diversity of each query. In the same table, we also present the original results in the FULL columns as a comparison.
We can observe that there is a significant performance gap between the FULL and DIVR results, which demonstrates that the diversified queries are harder to answer. Meanwhile, it is also observed that comparing to Q2P-1P, Q2P-KP  $(\mathrm{K} > 1)$  significantly improves the MRR of DIVR queries by 7.8 points. From this perspective, the improvement of Q2P-KP  $(\mathrm{K} > 1)$  over Q2P-1P is significant on those challenging queries.
5.6 Further Ablation Study for Q2P-1P
To better explain the superior performance of Q2P-1P over the baseline models, we conduct further ablations studies in Table 5.
First, we remove all the self-attention layers Attn. Then the performance of intersection operations largely decreased. This can be explained that the self-attention structure is important for aggregating the information from multiple sub-queries.
Then, we remove all the neural network structures, including all MLP and Attn from all operations, and replace them with the operations defined in the GQE model (Hamilton et al., 2018). Then
MODELS	1P	2I	2U	2IN	AVERAGE
FULL	DIVR	FULL	DIVR	FULL	DIVR	FULL	DIVR	FULL	DIVR
Q2P-1P	81.8	44.8	63.4	28.8	33.4	11.3	18.9	15.0	49.4	25.0
Q2P-2P	82.6	49.4	65.1	35.5	32.1	13.3	21.9	20.7	50.4	29.7
Q2P-3P	82.9	53.0	64.4	37.7	33.6	18.6	21.8	21.6	50.6	32.8
Table 4: The FULL columns demonstrate the averaged MRR results, and the DIVR columns demonstrate the averaged MRR on the top ten-percent diversified queries.
MODELS	1P	2P	2I	2U	2IN
Q2P-KP	83.4	31.5	66.0	38.9	22.3
Q2P-1P	81.8	30.7	63.4	33.4	18.9
- SELF ATTENTION	78.5	28.5	30.9	30.3	15.2
- ALL NNs + GQE Ops	56.7	16.1	39.2	20.1	-
Q2P	68.0	21.0	55.1	35.1	-
GQE	54.6	15.3	39.7	22.1	-
Table 5: The ablation study on the Q2P neural networks. The Q2P-KP result shows the highest result when K is ranged from 2 to 6.
the performance of Q2P is also reduced. This indicates that the neural structures in the particle operations are also important to the overall improvement. Thus, we infer that the baseline model underfit the complex queries in the training set, and the performance can be improved by introducing more parameters and non-linearity. This conclusion is also aligned with Sun et al. (2020), in which they found the baselines cannot faithfully answer the queries that are observed in the training time.
However, solely using more complex structures cannot address the problem raised from the diversity of the answers. As shown in Table 4, on the top of Q2P-1P, Q2P-KP  $(\mathrm{K} > 1)$  can still largely improve the performance on the diversified queries.
In this paper, we proposed Query2Particles, a query embedding method for answering complex logical knowledge graph queries over incomplete knowledge graphs. The Query2Particle method supports a full set of FOL operations. Specifically, the Q2P method is the first query embedding method that can directly model the union operation without any preprocessing. Experimental results show that the Q2P method achieves state-of-the-art performances on answering FOL queries on three different knowledge graphs while using comparable inference time as the previous methods.
7 Ethical Impacts
This paper introduces a knowledge graph reasoning method, and the experiments are on several publicly available benchmark datasets. As a result, there is
no data privacy concern. Meanwhile, this paper does not involve human annotations, and there is no related ethical concerns.
The authors of this paper were supported by the NSFC Fund (U20B2053) from the NSFC of China, the RIF (R6020-19 and R6021-20) and the GRF (16211520) from RGC of Hong Kong, the MHKJFS (MHP/001/19) from ITC of Hong Kong and the National Key R&D Program of China (2019YFE0198200) with special thanks to Hong Kong Mediation and Arbitration Centre (HKMAAC) and California University, School of Business Law & Technology (CUSBLT), and the Jiangsu Province Science and Technology Collaboration Fund (BZ2021065).
Erik Arakelyan, Daniel Daza, Pasquale Minervini, and Michael Cochez. 2020. Complex query answering with neural link predictors. In International Conference on Learning Representations.
Christian Bizer, Jens Lehmann, Georgi Kobilarov, Soren Auer, Christian Becker, Richard Cyganiak, and Sebastian Hellmann. 2009. Dbpedia-a crystallization point for the web of data. Journal of web semantics, 7(3):154-165.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of ACM SIGMOD 2008, pages 1247-1250.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26.
Andrew Carlson, Justin Betteridge, Bryan Kisel, Burr Settles, Estevam R Hruschka, and Tom M Mitchell. 2010. Toward an architecture for never-ending language learning. In Twenty-Fourth AAAI conference on artificial intelligence.
Xiaojun Chen, Shengbin Jia, and Yang Xiang. 2020. A review: Knowledge reasoning over knowledge graph. Expert Systems with Applications, 141:112948.
Xuelu Chen, Muhao Chen, Weijia Shi, Yizhou Sun, and Carlo Zaniolo. 2019. Embedding uncertain knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3363-3370.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, December 2014.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Shu Guo, Quan Wang, Lihong Wang, Bin Wang, and Li Guo. 2016. Jointly embedding knowledge graphs and logical rules. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 192-202.
Shu Guo, Quan Wang, Lihong Wang, Bin Wang, and Li Guo. 2018. Knowledge graph embedding with iterative guidance from soft rules. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Will Hamilton, Payal Bajaj, Marinka Zitnik, Dan Jurafsky, and Jure Leskovec. 2018. Embedding logical queries on knowledge graphs. Advances in neural information processing systems, 31.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2018. Multi-hop knowledge graph reasoning with reward shaping. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3243-3253.
Zhiyuan Liu, Maosong Sun, Yankai Lin, and Ruobing Xie. 2016. Knowledge representation learning: a review. Journal of Computer Research and Development, 53(2):247.
Hongyu Ren, Weihua Hu, and Jure Leskovec. 2020. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. In International Conference on Learning Representations.
Hongyu Ren and Jure Leskovec. 2020. Beta embeddings for multi-hop logical reasoning in knowledge graphs. Advances in Neural Information Processing Systems, 33:19716-19726.
Haitian Sun, Andrew Arnold, Tania Bedrax Weiss, Fernando Pereira, and William W Cohen. 2020. Faithful embeddings for knowledge base queries. Advances in neural information processing systems, 33.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2018. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations.
Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd workshop on continuous vector space models and their compositionality, pages 57-66.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International conference on machine learning, pages 2071-2080. PMLR.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28.
Robert West, Evgeniy Gabrilovich, Kevin Murphy, Shaohua Sun, Rahul Gupta, and Dekang Lin. 2014. Knowledge base completion via search-based question answering. In Proceedings of the 23rd international conference on World wide web, pages 515-526.
Wenhan Xiong, Thien Hoang, and William Yang Wang. 2017. Deeppath: A reinforcement learning method for knowledge graph reasoning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 564-573.
Zhanqiu Zhang, Jie Wang, Jiajun Chen, Shuiwang Ji, and Feng Wu. 2021. Cone: Cone embeddings for multi-hop reasoning over knowledge graphs. Advances in Neural Information Processing Systems, 34.
DATASET	RELATIONS	ENTITIES	TRAINING	VALIDATION	TESTING	ALL EDGES
FB15K	1,345	14,951	483,142	50,000	59,071	592,213
FB15K-237	237	14,505	272,115	17,526	20,438	310,079
NELL995	200	63,361	114,213	14,324	14,267	142,804
Table 6: The basic information about the three knowledge graph used for the experiments, and their standard training, validation, and testing edges separation according to (Ren and Leskovec, 2020).
DATASET	BATCH SIZE	DROPOUT RATIO	LABEL SMOOTHING	LEARNING RATE
FB15K	8,192	0.1	0.5	0.001
FB15K-237	8,192	0.1	0.5	0.001
NELL995	4,096	0.3	0.7	0.0003
Table 7: The best hyperparameters used by the Query2Particles model for the experiments on the queries originally used by (Ren and Leskovec, 2020).
(REN ET AL., 2020)	TRAINING	VALIDATION	TEST
DATASET	1P	OTHERS	1P	OTHERS	1P	OTHERS
FB15K	273,710	273,710	59,097	8,000	67,016	8,000
FB15K-237	149,689	149,689	20,101	5,000	22,812	5,000
NELL995	107,982	107,982	16,927	4,000	17,034	4,000
(REN AND LESKOVEC, 2020)	TRAINING	VALIDATION	TEST
DATASET	1P/2P/3P/2I/3I	2IN/3IN/INP/PIN/PNI	1P	OTHERS	1P	OTHERS
FB15K	273,710	273,71	59,097	8,000	67,016	8,000
FB15K-237	149,689	149,68	20,101	5,000	22,812	5,000
NELL995	107,982	107,98	16,927	4,000	17,034	4,000
Table 8: The detailed information for the queries used for training, validating, and testing all query embedding methods. The upper parts disclose the statistics of the queries taken from the (Ren et al., 2020) paper, while the lower part describes the queries taken from (Ren and Leskovec, 2020). The major differences are that the queries in (Ren and Leskovec, 2020) is harder than (Ren et al., 2020), and include five additional types of queries with the complement operation.
DATASET	MODEL	1P	2P	3P	2I	3I	Pi	IP	2U	UP	AVG
FB15K	BETAE	65.1	25.7	24.7	55.8	66.5	43.9	28.1	40.1	25.4	41.6
Q2B	68.0	21.0	14.2	55.1	66.5	39.4	26.1	35.1	16.7	38.0
GQE	54.6	15.3	10.8	39.7	51.4	27.6	19.1	22.1	11.6	28.0
Q2P (OURS)	82.6	30.8	25.5	65.1	74.7	49.5	34.9	32.1	26.2	46.8
FB15K-237	BETAE	39.0	10.9	10.0	28.8	42.5	22.4	12.6	12.4	9.9	20.9
Q2B	40.6	9.4	6.8	29.5	42.3	21.2	12.6	11.3	7.6	20.1
GQE	35.0	7.2	5.3	23.3	34.6	16.5	10.7	8.2	5.7	16.3
Q2P (OURS)	39.1	11.4	10.1	32.3	47.7	24.0	14.3	8.7	9.1	21.9
NELL	BETAE	53.0	13.0	11.4	37.6	47.5	24.1	14.3	12.2	8.6	24.6
Q2B	42.2	14.0	11.2	33.3	44.5	22.4	16.8	11.3	10.3	22.9
GQE	32.8	11.9	9.6	27.5	35.2	18.4	14.4	8.5	8.8	18.6
Q2P (OURS)	56.5	15.2	12.5	35.8	48.7	22.6	16.1	11.1	10.4	25.5
AVERAGE	BETAE	52.4	16.5	15.4	40.7	52.2	30.1	14.3	21.6	14.5	29.1
Q2B	50.3	14.8	10.7	39.3	51.1	27.7	18.5	19.2	11.5	27.0
GQE	40.8	11.5	8.6	30.2	40.4	20.8	14.7	12.9	8.7	21.0
Q2P (OURS)	59.4	19.1	16.0	44.7	57.0	32.0	21.8	17.3	15.2	31.3
Table 9: The MRR result for existential positive first order queries comparing to the BetaE, Q2B, and GQE methods. The results are reported from the queries used by Ren and Leskovec (2020).
DATASET	MODEL	1P	2P	3P	2I	3I	IP	PI	2U	UP	Avg
FB15K	EMQL	42.4	50.2	45.9	63.7	70.0	60.7	61.4	9.0	42.6	49.5
-SKETCH	50.6	46.7	41.6	61.8	67.3	54.2	53.5	21.6	40.0	48.6
CQD-BEAM	91.8	77.9	57.7	79.6	83.7	37.5	65.8	83.9	34.5	68.0
CQD-CO	91.8	45.4	19.1	79.6	83.7	33.6	51.3	81.6	31.9	57.6
Q2P (OURS)	90.2	74.6	73.4	86.0	89.6	63.7	77.6	83.4	52.7	76.8
FB15K-237	EMQL	37.7	34.9	34.3	44.3	49.4	40.8	42.3	8.7	28.2	35.8
-SKETCH	43.1	34.6	33.7	41.0	45.5	36.7	37.2	15.3	32.5	35.5
CQD-BEAM	51.2	28.8	22.1	35.2	45.7	12.9	24.9	28.4	12.1	29.0
CQD-CO	51.2	21.3	13.1	35.2	45.7	14.6	22.2	28.1	13.2	27.2
Q2P (OURS)	49.0	44.2	44.6	50.1	57.5	34.1	44.2	32.9	30.6	43.0
NELL	EMQL	41.5	40.5	38.6	62.9	74.5	49.8	64.8	12.6	35.8	46.8
-SKETCH	48.3	39.5	35.2	57.2	69.0	48.0	59.9	25.9	38.2	46.8
CQD-BEAM	66.7	35.0	28.8	41.0	52.9	17.1	27.7	53.1	15.6	37.6
CQD-CO	66.7	26.5	22.0	41.0	52.9	19.6	30.2	53.1	19.4	36.8
Q2P (OURS)	67.0	53.0	52.6	52.9	69.0	38.0	47.0	52.9	37.0	52.2
AVERAGE	EMQL	40.5	41.9	39.6	57.0	64.6	50.4	56.2	10.1	35.5	44.0
-SKETCH	47.3	40.3	36.8	53.3	60.6	46.3	50.1	20.9	36.9	43.6
CQD-BEAM	69.9	47.2	36.2	51.9	60.8	22.5	39.5	55.1	20.7	44.9
CQD-CO	69.9	31.1	18.1	51.9	60.8	22.6	34.6	54.3	21.5	40.5
Q2P (OURS)	68.7	57.3	56.9	63.0	72.0	45.3	56.3	56.4	40.1	57.3
Table 10: The Hit@3 results for existential positive first order queries comparing to the EmQL and CQD method over the queries used by Ren et al. (2020).