SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The RF-IEF matrix  $\pmb{F}$  captures both the local and global KG structure and it can be further enhanced with the rich semantic information on the edges of KGs. Unlike entity nodes with uninformative text, e.g., proper nouns and encrypted entity names, edges generally are accompanied by linguistic descriptions (relation types). Hence, the relations are commonly embedded by a pre-trained language model in KGQA. Combining with the relation embeddings  $\pmb{R} \in \mathbb{R}^{|\mathcal{R}| \times D}$ , our final RF-IEF node embeddings  $\pmb{H}$  is computed as
$$
\boldsymbol {H} = \boldsymbol {F} \boldsymbol {R} \boldsymbol {W} _ {h}, \tag {15}
$$
where  $\pmb{H} \in \mathbb{R}^{|\mathcal{V}_q| \times D}$ , and  $\pmb{W}_h \in \mathbb{R}^{D \times D}$ . The RF-IEF node embeddings can be viewed as the aggregated semantics of relations, which are represented by a language model, based on graph topology as well as the informativeness (or rareness) of relations. A row in  $\pmb{H}$  is a node embedding vector  $\mathbf{h}_\nu$ , which is used in Eq. (5) at the first NuTrea layer.
Expressiveness of the NuTrea layer. While many previous multi-hop KGQA models simultaneously update all node embeddings, recent works [14, 2, 1] have shown the superiority of approaches that search paths on the KG. These methods gradually expand the searching area by sequentially
updating nodes closer to the seed node towards the answer nodes. Our model builds on the latter sequential search scheme as well, enhancing expressiveness with our proposed NuTrea layers.
With a simple toy example in Figure 2, we compare the message flow of previous sequential search models and our NuTrea. The example demonstrates that our NuTrea layer (b) can probe subtrees to quickly gather fringe node (i.e., node 4 or 5) information without exhaustively visiting them. This is accomplished by our Backup step, which boosts the original past-oriented node embeddings with future-oriented subtree information.
Datasets. We experiment on three large-scale multi-hop KGQA datasets: MetaQA [35], WebQuestionsSP (WQP) [16] and ComplexWebQuestions (CWQ) [17]. Meta-QA consists of three different splits, 1-hop, 2-hop, and 3-hop, each indicating the number of hops required to reach the answer node from the seed node. The questions are relatively easy with less constraints. WQP and CWQ, on the other hand, contain more complex questions with diverse constraints. WQP is relatively easier, as CWQ is derived from WQP by extending its questions with additional constraints. MetaQA is answerable with the WikiMovies knowledge base [27], while WQP and CWQ require the Freebase KG [15] to answer questions. Further dataset information and statistics are provided in the supplement.
Baselines. We mainly compare with previous multi-hop KGQA methods that take the Information Retrieval approach (section 2). These models, unlike Semantic Parsing approaches, do not access the ground truth logical queries and focus on processing the KG subgraph to rank the nodes to identify answer nodes. To introduce the three most recent baseline models: (1) SQALER [3] proposes a scalable KGQA method whose complexity scales linearly with the number of relation types, rather than nodes or edges. (2) TERP [2] introduces the rotate-and-scale entity link prediction framework to integrate textual and KG structural information. (3) ReaRev [1] adaptively selects the next reasoning step with a variant of breadth-first search (BFS). Other baselines are introduced in the supplement.
Implementation Details. For WQP, 2 NuTrea layers with subtree depth  $K = 1$  is used, while CWQ with more complex questions uses 3 layers with depth  $K = 2$ . In the case of RF-IEF node embedding, we pre-compute the Entity Frequency (EF) values in Eq. (13) for subgraphs in the training set before training. We use the same EF values throughout training, validation, and testing. This stabilizes computation by mitigating the large variance induced by relatively small batch sizes. For MetaQA, the number of NuTrea layers are selected from  $\{2, 3\}$ , and  $K$  for ego-graph pooling from  $\{1, 2\}$ . See the supplement for further hyperparameter settings and details.
4.2 Main Experiments
Here, we present the experimental results of NuTrea. Following the common evaluation practice of previous works, we test the model that achieved the best performance on the validation set. In the WQP dataset experiments in Table 1, we achieved the best performance of  $77.4\mathrm{H}@\mathbf{1}$  among strong KGQA baselines that take an information retrieval approach, as discussed in Section 2. Compared to the previous best, this is a large improvement of 0.6 points. In terms of the F1 score, which evaluates the answer set prediction, our method achieved a score of 72.7, exceeding the previously recorded value by a large margin of 1.8 points. In addition, we also improved the previous state-of-the-art performance on the CWQ dataset by achieving an  $53.6\mathrm{H}@\mathbf{1}$ , which is an improvement of 0.7 points.
We also experimented NuTrea on MetaQA to see if it performs reasonably well with easy questions as well. On three data splits, NuTrea achieved comparable performance with previous state-of-the-art methods for simple question answering. Evaluating with the average H@1 score of the three splits, NuTrea performs second best among all baseline models.
4.3 Incomplete KG Experiments
The KG is often human-made and the contents are prone to being incomplete. Hence, it is a norm to test a model's robustness on incomplete KG settings where a certain portion of KG triplets, i.e., a tuple of ⟨head, relation, tail⟩, are dropped. This experiment evaluates the robustness of our model to missing relations in a KG. We follow the experiment settings in [1], and use the identical incomplete KG dataset which consists of WQP samples with  $[50\%, 30\%, 10\%]$  of the original KG triplets remaining. In Table 2, NuTrea performs the best in most cases, among the GNN models
Table 1: Results on multi-hop KGQA datasets. The Hit@1 and F1 scores are reported. The baselines are taken from the original papers. The best performances are in bold, and the second best are underlined.
Models	WQP	CWQ	MetaQA
H@1	F1	H@1	F1	1-hop	2-hop	3-hop	Avg. H@1
KV-Mem [27]	46.7	38.6	21.1	-	95.8	25.1	10.1	43.7
GraftNet [28]	66.7	62.4	32.8	-	-	-	-	96.8
PullNet [29]	68.1	-	45.9	-	97.0	99.9	91.4	96.1
EmbedKGQA [11]	66.6	-	-	-	97.5	98.8	94.8	97.0
ReifiedKB [36]	52.7	-	-	-	96.2	81.1	72.3	83.2
EMQL [37]	75.5	-	-	-	97.2	98.6	99.1	98.3
TransferNet [13]	71.4	48.6	48.6	-	97.5	100.0	100.0	99.2
NSM(+p) [14]	73.9	66.2	48.3	44.0	97.3	99.9	98.9	98.7
NSM(+h) [14]	74.3	67.4	48.8	44.0	97.2	99.9	98.9	98.6
Rigel [38]	73.3	-	48.7	-	-	-	-	-
SQALER+GNN [3]	76.1	-	-	-	-	99.9	99.9	-
TERP [2]	76.8	-	49.2	-	97.5	99.4	98.9	98.6
ReaRev [1]	76.4	70.9	52.9	-	-	-	-	-
NuTrea (Ours)	77.4	72.7	53.6	49.5	97.4	100.0	98.9	98.8
Table 2: Incomplete KG experiments. NuTrea also performs well in incomplete KG settings. The baseline figures were taken from [1].
Portion of KG triplets (%)	50%	30%	10%
H@1	F1	H@1	F1	H@1	F1
Graftnet [28]	47.7	34.3	34.9	20.4	15.5	6.5
SGReader [39]	49.2	33.5	35.9	20.2	17.1	7.0
HGCN [40]	49.3	34.3	35.2	21.0	18.3	7.9
ReaRev [1]	53.4	39.9	37.9	23.6	19.4	8.6
NuTrea (Ours)	53.7	40.1	38.3	24.1	18.9	8.7
Table 3: Component ablation experiments. The ablation experiments were done on the WQP dataset. Two main contributions are studied.
RF-IEF
Node Emb.	Backup
step	WQPH@1	WQPF1
✓	✓	77.4 (−0.0)	72.7 (−0.0)
✓	74.8 (−2.6)	70.4 (−2.3)
✓	76.8 (−0.6)	71.5 (−1.2)
73.4 (−4.0)	70.9 (−1.8)
designed to handle incomplete KGs. We believe that our model adaptively learns multiple alternative reasoning processes and can plan for future moves beforehand via our Backup step, so that it provides robust performance with noisy KGs.
In this section, we provide comprehensive analyses on the contributions of NuTrea to ensure its effectiveness in KGQA. We try to answer the following research questions: Q1. How does each component contribute to the performance of NuTrea? Q2. What is the advantage of NuTrea's tree search algorithm over previous methods? Q3. What are the effects of the RF-IEF node embeddings?
5.1 Ablation Study
Here, we evaluate the effectiveness of each component in NuTrea to answer Q1. An ablation study is performed on our two major contributions: the Backup step in our NuTrea layers and the RF-IEF node embeddings. By removing the RF-IEF node embeddings, we instead apply the common node initialization method used in [14, 1], which simply averages the relation embeddings incident to each node. Another option is to use zero embeddings, but we found it worse than the simple averaging method. For NuTrea layer ablation, we remove the Backup step which plays a key role in aggregating the future-oriented subtree information onto the KG nodes. Then, only the expansion-score  $(s_{\nu}^{(e)})$  is computed, and the backup-score  $(s_{\nu}^{(b)})$  is always 0. Also, the embedding  $\mathbf{f}_{\nu}$  (Eq. (5)), is directly output from the NuTrea layer and no further updates are made via the Backup step.
In Table 3, we can see that the largest performance drop is observed when the Backup step is removed. Without it, the model has limited access to the broader context of the KG and cannot reflect the complex question constraints in node searching. Further discussion on this property of NuTrea's message passing is provided in the next Section 5.2. Also, we observed a non-trivial 0.6 point drop
Figure 3: Qualitative examples (without Backup  $\Rightarrow$  with Backup) (section 5.2.2).
Figure 4: Weights computed by RF-IEF (section 5.3).
in H@1 by removing the RF-IEF initialization method. By ablating both components, there was a significant degradation of  $4.0\mathrm{H}@\mathbf{1}$  points.
5.2 Advantages of NuTrea
To answer Q2, we highlight the key advantages of NuTrea over recent approaches. We analyze the efficiency of NuTrea, and provide qualitative results.
5.2.1 Efficiency of NuTrea
In addition, to further verify the utility of our model, we provide analyses on the latency of NuTrea with WQP in Table 4. Compared to the most recent ReaRev [1] model, training and inference latency per epoch/sample is slightly bigger, due to our additional Backup module. However,
thanks to our expressive message passing scheme, NuTrea converges way faster, allowing the training GPU hours to be reduced from 4.3 hours to 2.9 hours.
To provide more insight in terms of number of NuTrea layers, we also reveal its effect on model performance, in Figure 5. The figure reports the F1 scores for both with and without the Backup module, evaluated on the WebQuestionsSP dataset. "NuTrea without Backup" has an equivalent model configuration used in the Backup ablation experiment of Table 3. Overall, the performance of "NuTrea without Backup" does generally improve with additional layers, but "NuTrea with Backup" reached the highest score of 72.7 with only 2 NuTrea layers. This is enabled by our Backup module, which alleviates the burden of exhaustively searching deeper
Table 4: Latency of ReaRev and NuTrea.
Models	NuTrea (Ours)	ReaRev
Training Latency (per epoch)	100.2 s	78.0 s
Inference Latency (per sample)	67.7 ms	51.3 ms
Training GPU Hours	2.9 H	4.3 H
Figure 5: Effect of number of layers.
nodes. This is computationally more efficient than stacking multiple layers to achieve higher performance. Specifically, by comparing the 2-layer "NuTrea with Backup" and 5-layer "NuTrea without Backup", the former required an average of  $73.8\mathrm{ms}$  of inference time per question, whereas the latter required  $108.6\mathrm{ms}$ . With only  $68\%$  of compute, our NuTrea achieved comparable performance with the deeper "NuTrea without Backup". Note, these latency values were evaluated on a different environment from values reported in Table 4.
5.2.2 Qualitative Results
In Figure 3, we demonstrate several qualitative examples from our error analysis. In each question, the blue (node) entity is the correct answer choice, while the red is the wrong choice made by the
model without the Backup step. The values in parentheses demonstrate the difference in node scores between models without and with Backup. Without it, the sequential search model cannot refer to local contexts of a node and frequently predicts an extremely low score for the correct answer node (e.g., 0.0062 for "Queen Elizabeth the Queen Mother" in the first question of Figure 3). Such a problem is mitigated by our NuTrea's Backup step, which tends to boost scores of correct answers and tone down wrong choices that were wrongly assigned with a high score. More qualitative examples are provided in the supplement, along with an analysis on the different importances of the Backup step in different datasets, by controlling the context coefficient  $\lambda$ .
5.3 Effect of RF-IEF
The RF-IEF node embedding is a simple method inspired by an effective text representation technique in natural language processing. Here, we disclose the specific effect of RF-IEF on the relation embedding aggregation weights, thereby answering Q3. In Figure 4 (left), the log-scaled  $EF$  (Eq. 13) value of each relation type is sorted. The globally most frequent relation types, including "self_loop"s, are too general to provide much context in characterizing a KG node. Our RF-IEF suppresses such uninformative relation types for node embedding initialization, resulting in a weight distribution like Figure 4 (right). The pie charts display two examples on the difference between aggregation weights for a node entity before and after RF-IEF is applied. To illustrate, the weight after RF-IEF corresponds to a row of  $F$  in Eq. (14), while the weight before RF-IEF would be uniform across incident edges. As "Alexander Bustamante" is a politician, the relation types "organizations founded" and "founders" become more lucid via RF-IEF, while relations like "events_competed_in" and "competitors" are emphasized for an athlete like "Kemar Bailey-Cole". Likewise, RF-IEF tends to scale up characteristic relation types in initializing the node features, thereby enhancing differentiability between entities. To further demonstrate RF-IEF's general applicability, we also provide a plug-in experiment on another baseline model in the supplement.
Neural Tree Search (NuTrea) is an effective GNN model for multi-hop KGQA, which aims to better capture the complex question constraints by referring to the broader KG context. The high expressiveness of NuTrea is attained via our message passing scheme that resembles the MCTS algorithm, which leverages the future-oriented subtree information conditioning on the question constraints. Moreover, we introduce the RF-IEF node embedding technique to also consider the global KG context. Combining these methods, our NuTrea achieves the state-of-the-art in two major multi-hop KGQA benchmarks, WebQuestionsSP and ComplexWebQuestions. Further analyses on KG incompleteness and the qualitative results support the effectiveness of NuTrea. Overall, NuTrea reveals the importance of considering the broader KG context in harnessing the knowledge graph via human languages.
This work was partly supported by ICT Creative Consilience program (IITP-2023-2020-0-01819) supervised by the IITP, the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2023R1A2C2005373), and KakaoBrain corporation.
[1] Costas Mavromatis and George Karypis. Rearev: Adaptive reasoning for question answering over knowledge graphs. In EMNLP-Findings, 2022.
[2] Zile Qiao, Wei Ye, Tong Zhang, Tong Mo, Weiping Li, and Shikun Zhang. Exploiting hybrid semantics of relation paths for multi-hop question answering over knowledge graphs. In COLING, 2022.
[3] Mattia Atzeni, Jasmina Bogojeska, and Andreas Loukas. Sqaler: Scaling question answering by decoupling multi-hop and logical reasoning. NeurIPS, 2021.
[4] Jinyoung Park, Hyeong Kyu Choi, Juyeon Ko, Hyeonjin Park, Ji-Hoon Kim, Jisu Jeong, Kyungmin Kim, and Hyunwoo Kim. Relation-aware language-graph transformer for question answering. In AAAI, 2023.
[26] Rajarshi Das, Ameya Godbole, Ankita Naik, Elliot Tower, Manzil Zaheer, Hannaneh Hajishirzi, Robin Jia, and Andrew McCallum. Knowledge base question answering by case-based reasoning over subgraphs. In ICML. PMLR, 2022.
[27] Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. Key-value memory networks for directly reading documents. In EMNLP, 2016.
[28] Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. Open domain question answering using early fusion of knowledge bases and text. In EMNLP, 2018.
[29] Haitian Sun, Tania Bedrax-Weiss, and William Cohen. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. In EMNLP-IJCNLP, 2019.
[30] Lihui Liu, Boxin Du, Jiejun Xu, Yinglong Xia, and Hanghang Tong. Joint knowledge graph completion and question answering. In ACM SIGKDD, 2022.
[31] Scott Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of the Joint Conference of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing of the AFNLP, 2015.
[32] Hans Peter Luhn. The automatic creation of literature abstracts. IBM Journal of research and development, 2(2):159-165, 1958.
[33] Stephen E Robertson and K Sparck Jones. Relevance weighting of search terms. Journal of the American Society for Information science, 27(3):129-146, 1976.
[34] Gerard Salton and Christopher Buckley. Term-weighting approaches in automatic text retrieval. Information processing & management, 24(5):513-523, 1988.
[35] Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J Smola, and Le Song. Variational reasoning for question answering with knowledge graph. In AAAI, 2018.
[36] William W Cohen, Haitian Sun, R Alex Hofer, and Matthew Siegler. Scalable neural methods for reasoning with a symbolic knowledge base. In ICLR, 2020.
[37] Haitian Sun, Andrew Arnold, Tania Bedrax Weiss, Fernando Pereira, and William W Cohen. Faithful embeddings for knowledge base queries. NeurIPS, 2020.
[38] Priyanka Sen, Armin Oliya, and Amir Saffari. Expanding end-to-end question answering on differentiable knowledge graphs with intersection. In EMNLP, 2021.
[39] Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. Improving question answering over incomplete kbs with knowledge-aware reader. ACL, 2019.
[40] Jiale Han, Bo Cheng, and Xu Wang. Open domain question answering based on text enhanced knowledge graph with hyperedge infusion. In EMNLP-Findings, 2020.