Reasoning over Hierarchical Question Decomposition Tree for Explainable Question Answering
Year: 2023

Abstract:
Explainable question answering (XQA) aims to answer a given question and provide an explanation why the answer is selected. Existing XQA methods focus on reasoning on a single knowledge source, e.g., structured knowledge bases, unstructured corpora, etc. However, integrating information from heterogeneous knowledge sources is essential to answer complex questions. In this paper, we propose to leverage question decomposing for heterogeneous knowledge integration, by breaking down a complex question into simpler ones, and selecting the appropriate knowledge source for each sub-question. To facilitate reasoning, we propose a novel two-stage XQA framework, Reasoning over Hierarchical Question Decomposition Tree (RoHT). First, we build the Hierarchical Question Decomposition Tree (HQDT) to understand the semantics of a complex question; then, we conduct probabilistic reasoning over HQDT from root to leaves recursively, to aggregate heterogeneous knowledge at different tree levels and search for a best solution considering the decomposing and answering probabilities. The experiments on complex QA datasets KQA Pro and Musique show that our framework outperforms SOTA methods significantly, demonstrating the effectiveness of leveraging question decomposing for knowledge integration and our RoHT framework.

heading: 1 Introduction
text: Explainable question answering (XQA) is the task of (i) answering a question and (ii) providing an explanation that enables the user to understand why the answer is selected (Neches et al., 1985; Schuff et al., 2020). It provides a qualified way to test the reasoning ability and interpretability of intelligent systems, and plays an important role in artificial intelligence (Lu et al., 2022).
∗Indicates equal contribution. †Corresponding author.
Recent work in XQA can be grouped into two directions: 1) neuro-symbolic methods (Berant et al., 2013; Liang et al., 2017; Cao et al., 2022b) translate natural language questions into formal representations (e.g., SPARQL (Sun et al., 2020), KoPL (Cao et al., 2022a), lambda-DCS (Liang, 2013), etc.), whose execution on structured knowledge bases (KBs) gives the answer. Here, the formal representation acts as an explanation of the final answer. 2) Decompose-based models generate natural language intermediate steps that lead to the final answer (e.g., question decomposing which decomposes a complex question into sub-questions (Min et al., 2019; Perez et al., 2020; Deng et al., 2022), chain-of-thought prompting (Wei et al., 2022; Dua et al., 2022; Khot et al., 2022), etc.). Here, the intermediate steps shows the rationale of reasoning.
Although achieving significant results, both directions have key limitations. For neuro-symbolic methods, the formal representation can only be executed on KBs. However, even the largest KBs are incomplete, thus limits the recall of model. For decompose-based methods, they employ free-text corpora as the knowledge source, and the diversity of natural language makes XQA difficult. In fact, integrating knowledge from heterogeneous sources is of great importance to QA (Wolfson et al., 2020), especially for answering complex questions. Several attempts have been made for knowledge integration (e.g., KBs, text corpora) (Sun et al., 2018,
14556
2019; Shi et al., 2021). Although promising, these graph-based methods suffer from lacking explainability or are constrained to limited reasoning capability.
Intuitively, leveraging question decomposing to integrate heterogeneous knowledge sources is a promising direction, since we can flexibly select the appropriate knowledge source for each subquestion. The challenges lie in: 1) How to determine the granularity of question decomposing, since certain complex questions can be directly answered with a knowledge source, and further decomposition increases the possibility of error. For example, in Figure 1, q1 can be answered with the Wikipedia corpus without further decomposition. 2) How to find the optimal solution among various possible ones, since question decomposing and answering are both uncertain. For example, q0 can also be decomposed as “Which mountains are in North America or Afirica”, “What’s the height of #1”, “[SelectAmong] [largest] #2”.
To this end, we propose a novel two-stage XQA framework Reasoning over Hierarchical Question Decomspotion Tree, dubbed RoHT. First, we propose to understand the complex question by building its hierarchical question decomposition tree (HQDT). In this tree, the root node is the original complex question, and each non-root node is a subquestion of its parent. The leaf nodes are atomic questions that cannot be further decomposed. Compared with existing representations that directly decompose a question into the atomic ones, e.g., QDMR (Wolfson et al., 2020), our tree structure provides the flexibility to determine solving a question whether by directly answering or further decomposing. Second, we propose probabilistic reasoning over HQDT, to fuse the knowledge from KB and text at different levels of the tree, and take into consideration the probability score of both tree generation and answering. The reasoning process is recursive, from the root to leaves, and constitues three steps: 1) a scheduler determines the appropriate knowledge sources for a particular question (from KB, text, or solving its children sequentially); 2) the corresponding executors output the answers with probabilities; 3) an aggregator aggregates the candidate answers from all the knowledge sources and outputs the best ones.
In evaluation, we instantiate our RoHT framework on two complex QA datasets: KQA Pro (Cao et al., 2022a), where we remove half of the triples
in its KB and supplement it with Wikipedia corpus, and Musique (Trivedi et al., 2022), where we take Wikidata (Vrandecic and Krötzsch, 2014) as additional KB besides the given text paragraphs. Experimental results show that, RoHT improves the performance significantly under the KB+Text setting, by 29.7% and 45.8% EM score on KQA Pro and Musique compared with existing SOTA model. In addition, compared with the decompose-based methods, RoHT improves the SOTA by 11.3% F1 score on Musique.
Our contributions include: 1) proposing to leverage question decomposing to integrate heterogeneous knowledge sources for the first time; 2) designing a novel two-stage XQA famework RoHT by first building HQDT and then reasoning over HQDT; 3) demonstrating the effectiveness of our RoHT framework through extensive experiments and careful ablation studies on two benchmark datasets.

heading: 2 Related Work

heading: 2.1 QA over Text and KB
text: Over time, the QA task has evolved into two main streams: 1) QA over unstructured data (e.g., freetext corpora like Wikipedia); 2) QA over structured data (e.g., large structured KBs like DBpedia (Lehmann et al., 2015), Wikidata (Vrandecic and Krötzsch, 2014)). As structured and unstructured data are intuitively complementary information sources (Oguz et al., 2022), several attempts have been made to combines the best of both worlds.
An early approach IBM Watson (Ferrucci, 2012) combines multiple expert systems and re-ranks them to produce the answer. (Xu et al., 2016) maps relational phrases to KB and text simultaneously, and use an integer linear program model to provide a globally optimal solution. Universal schema based method (Das et al., 2017) reasons over both KBs and text by aligning them in a common embedded space. GraftNet (Sun et al., 2018) and its successor PullNet (Sun et al., 2019) incorporate free text into graph nodes to make texts amenable to KBQA methods. TransferNet (Shi et al., 2021) proposes the relation graph to model the label-form relation from KBs and text-form relation from corpora uniformly.
Although achieving promising results, these methods lack interpretability or are constrained to limited question type, i.e., TransferNet shows
interpretability with transparent step transfering, however, it can only answer multi-hop questions, and cannot deal with questions that require attribute comparison or value verification. In contrast, our proposed framework shows great interpretability with HQDT and cover more question types.

heading: 2.2 Question Decomposing
text: For datasets, KQA Pro (Cao et al., 2022a) proposes to decompose a complex question into a multi-step program KoPL, which can be executed on KBs. BREAK (Wolfson et al., 2020) proposes to decompose questions into QDMR, which constitutes the ordered list of steps, expressed through natural language. Musique (Trivedi et al., 2022) is a QA dataset constructed by composing single-hop questions obtained from existing datasets, and thus naturally provides question decompositions.
For models, several attempts have been made for learning to decompose with weak-supervision, such as span prediction based method (Min et al., 2019), unsupervised sequence transduction method ONUS (Perez et al., 2020), AMR-based method QDAMR (Deng et al., 2022). Another line of work is to employ large language models with in-context learning, such as Least-to-most Prompting (Zhou et al., 2022), decomposed prompting (Khot et al., 2022), successive prompting (Dua et al., 2022).
Compared with existing works, we are the first to design a hierarchical question decomposition tree for integrating information from multiple knowledge sources.

heading: 3 Definition of HQDT
text: Formally, given a complex question, its HQDT is a tree T . Each node qi ∈ T represents a question. For root node, it represents the given complex question, and for non-root nodes, it represents a sub-question of its parent node. The leaf nodes are simple ("atomic") questions that cannot be decomposed. Note that HQDT is a 3-ary ordered tree. As shown in Figure 1, we enumerate the nodes of T with BFS ordering, and q0 is the root question.
A question qi = 〈 w1, · · · , wj , · · · , w|qi| 〉 can be categorized into one of the three types according to the token vocabulary: 1) natural language question (e.g., q4: “Which mountain is the highest in North America?”), here, wj ∈ V , and V is the word vocabulary; 2) bridge question (e.g., q5: “How high is #4?”), here, wj ∈ V ∪ R, and R is the reference token vocabulary. In this ques-
tion, “#4” refers to the answer of q4, which is the sibling question of q5; 3) symbolic operation question (e.g., q3: “[SelectBetween][greater] #1 #2”), here, wj ∈ V ∪ R ∪ O, and O is the vocabulary of pre-defined symbolic operations, which are designed for supporting various reasoning capacity (e.g., attribute comparison and set operation) and are shown in appendix A in details. Note that all the bridge questions and symbolic operation questions are atomic questions and can only appear in leaf nodes.
For every non-leaf question qi, we define two ordered lists:
• qi.children = 〈 qsti , · · · , qedi 〉 , which are
children of qi, successively indexed from sti to edi. For example, for question q1 in Figure 1, q1.children is 〈 q4, q5 〉 .
• qi.atoms = 〈 ai1, · · · , aini 〉 , which is a list
of atomic questions deduced from the ni leaf nodes of the sub-tree rooted by qi, by rearranging the reference tokens. For example, for q0 in Figure 1, its leaf nodes is 〈 q4, q5, q6, q7, q3 〉 , and correspondingly, q0.atoms is 〈 q4, q̃5, q6, q̃7, q̃3 〉 , with q̃5 as “How high is #1?”, q̃7 as “How high is #3”, and q̃3 as “[SelectBetween][greater] #2 #4”. The detailed deduction algorithm is in appendix B due to space limit. We also call qi.atoms the atomic representation of qi.
Specially, among qi.children, qsti , . . . , qedi−1 are all natural language questions, and qedi is either a bridge question or a symbolic operation question. Answering qi is semantically equivalent to answering sub-questions in qi.children or in qi.atoms sequentially. The last question in qi.children or qi.atoms returns the answer of qi.

heading: 4 Methodology
text: Our framework RoHT is composed of two stages: 1) Building HQDT. We understand the hierarchical compositional structure of a complex question q0 by generating its HQDT T with probability, where each question qi ∈ T has a score pig that represents the certainty of its generation.
2) Probabilistic Reasoning over HQDT. We conduct recursive probabilistic reasoning over the HQDT from root to leaves to solve q0. For each question qi, we will utilize KBs, text and its child questions together to get a list Ri, which contains
answers of qi with probabilistic scores. Finally the answer with the highest score in R0 will be picked out as the final answer of q0.
The details are introduced as follows.

heading: 4.1 Building HQDT
text: To build the HQDT for a complex question, we first generate its atomic representation, which corresponds the leaf nodes of HQDT, then generate every non-leaf nodes based on this atomic representation. We compute certainty score of each node based on the likelihood of each step of generation.
Building Leaf Nodes Given a complex question q0, we first use a BART (Lewis et al., 2020)-based question decomposer Mθ to generate its atomic representation and output the likelihood of generation:
L0, ld = Mθ(q 0). (1)
Here, L0 = a01 ⟨sep⟩ a02 ⟨sep⟩ . . . ⟨sep⟩ a0n0 is the serialization of q0.atoms, where ⟨sep⟩ is a separating token. ld = Pr(L0|q0; θ) is the likelihood of generation. Since q0 is the root of T , each atomic question in q0.atoms corresponds to a leaf node in T (with the deterministic algorithm in Appendix C), and the certainty score of each leaf node in T is ld.
Building Non-leaf Nodes Based on q0.atoms, we can generate all the non-leaf questions in HQDT. The root question is just q0 and thus has certainty score p0g = 1. For every other non-leaf question qi, its atomic representation qi.atoms = ⟨ai1, . . . , aini⟩ can be translated from a specific subset of q0.atoms by rearranging the reference tokens. The subset can be determined by considering the reference relations of a bridge or symbolic operation question a0j ∈ q0.atoms, which corresponds to the leaf node qedi , with other questions in q0.atoms. We show the details in Appendix C. For example, q2.atoms in Figure 1 is (“Which mountain is the highest in Africa?”, “How high is #1?”), and it can be obtained from (a03, a 0 4) in q
0.atoms. Then we can use a BART-based question generator Mϕ to generate qi from qi.atoms:
qi, lig = Mϕ(L i), (2)
where Li = ai1 ⟨sep⟩ ai2 ⟨sep⟩ . . . ⟨sep⟩ aini is the serialized qi.atoms, and lig = Pr(q
i|Li;ϕ) is the likelihood of qi given Li. The certainty score of qi is computed as:
pig = ld · lig. (3)
Learning of Question Decomposer and Generator The question decomposer Mθ can be trained with paired (q0, q0.atoms) data, where the atomic representation can be from either given annotation or unsupervised construction. The question generator Mϕ can also be trained with the same data by exchanging the input and output. The details are shown in Section 5.2.

heading: 4.2 Probabilistic Reasoning over HQDT
text: f(qi, pig, G,C) → Ri : {(ansij , pij)}, (4)
where ansij is an answer of q i, and score pij represents the certainty of ansij . As shown in Figure 3, the implementation of f contains tree steps: 1) a scheduler determines the suitable knowledge sources for a particular question, i.e., whether the question can be answered from KB, text, or by solving its child questions sequentially; 2) according to the suitable sources output by the scheduler, executors aim to get the answers with probabilities via executing on KB (KB executor) or retrieving from text (text executor), or answering the child questions (call f recursively); 3) an aggregator aggregates candidate answers from all the knowledge sources and outputs the top-k answers according to their probabilities. In the following, we will introduce their details when answering qi.
Scheduler We formalize the scheduler as:
suitkb, suittext, suitchild = Scheduler(q i, G,C), (5) Where suitkb, suittext and suitchild are 0/1 variables, respectively representing whether the answers of qi are suitable to get from the KB G, the corpus C, or by solving qi.children sequentially.
Specifically, to check whether G is suitable, the scheduler employs a semantic parser (Cao et al., 2022a) Msp to parse qi into a program K with probability pparse:
K, pparse = Msp(q i). (6)
Then it classifies the type of qi according to the function skeleton of K. For example, the function skeleton of K in Figure 2 is “Find-RelateFilterConcept-SelectAmong”. If the precision of G on the questions that have the same function skeleton with K is larger than a predefined threshold γ 1, the scheduler will set suitkb to be 1.
1The precision of KB is calculated with questions in training set
To check whether the corpus C is suitable, the scheduler tries to find a set of evidence paragraphs for qi. If C is too large, the scheduler will first use BM25 (Robertson and Zaragoza, 2009) to recall dozens of most relevant paragraphs. For each paragraphs, we train a RoBERTa (Liu et al., 2019)- based selector Msl to classify whether it is an evidence paragraph for qi. Suppose the set of selected evidence paragraphs, Ce is not empty, the scheduler will set suittext as 1.
To make best use of knowledge from all levels, the scheduler simply set suitchild to be 1 if qi is a non-leaf question otherwise 0.
Executors For the KB executor, it takes the program K in Equation 6 on KB G to get the answers, and takes the parsing score pparse in Equation 6 to calculate the probability score for each answer:
Rikb = {(ansikb,j , pig · pparse)}. (7)
For the text executor, it takes the selected paragraph set Ce as described above, and employs a Transformer-based reading comprehension model Mrc to extract answers from Ce:
{(ansitext,j , piex,j)} = Mrc(qi, Ce), Ritext = {(ansitext,j , pig · piex,j)}.
(8)
where piex,j is the extraction probability of ans i text,j given by Mrc. For solving qi by answering its children, f will recursively call itself to solve qsti , . . . , qedi in or-
der:
Rsti = f(qsti , pstig , G,C), Rsti+1 = f(qsti+1, psti+1g , G,C), (9)
. . .
Redi = fref(q edi , pedig , G,C, [R
sti , . . . , Redi−1]),
and let Richild = R edi . (10)
Here, fref is a variant of f to solve bridge and symbolic questions, which refer to the answers of their sibling questions. Suppose qedi refers to the answers of its siblings qr1 , . . . , qrhi in order. If qedi is a bridge question, fref will 1) convert qedi into several possible natural language question q1nl, . . . , q K nl by replacing the reference tokens with every combination ((xk1, v k 1 ), . . . , (x k hi , vkhi)) ∈ R
r1 × · · · × Rrhi , 2) call f to solve each qknl and 3) fuse the answers from each Rknl and select the top-k answers with the highest scores:
{(ansknl,j , pknl,j)} = f(qjnl, pig, G,C), Rknl = {(ansknl,j , Avg(pknl,j , vk1 , . . . , vkhi))}, Redi = Select(R1nl, . . . , R K nl ) (11)
Note that the score of answer ansknl,j is computed by averaging pknl,j and v k 1 , . . . , v k hi
, instead of multiplying them, to avoid exponential shrink during recursion. If qedi is a symbolic operation question with operation op and arguments, fref will execute simple program to apply the operation op over Rr1 , . . . , Rrhi to get Redi . The score of each answer ansedij is computed as the average of p edi g and the scores of answers in Rr1 , . . . , Rrhi used by the program to get ansedij .
Aggregator The aggregator fuses Rikb, R i text and Richild by selecting the top-k answers with the highest scores from them. If several answers have the same surface form, only the one with the highest score will be preserved.
Ri = Aggregator(Rikb, R i text, R i child). (12)

heading: 5 Experiments

heading: 5.1 Datasets
text: Currently, there are few high-quality complex QA datasets based on both KBs and text. Previous methods (Sun et al., 2018, 2019; Shi et al., 2021) evaluated their models on MetaQA (Zhang et al., 2018) by pairing its KB with the text corpus of WikiMovies (Miller et al., 2016). However, the questions in MetaQA are too simple since there are only 9 relations in its KB. Therefore, we conduct our experiments on two more challenging complex QA datasets: KQA Pro and Musique, and their details are as follows.
KQA Pro (Cao et al., 2022a) is a large scale complex QA dataset, including 120k diverse natural language questions up to 5 hops over KB. Its KB is a subset of Wikidata (Vrandecic and Krötzsch, 2014), and consists of 16k entities, 363 predicates, 794 concepts and 890k triple facts. For each question, KQA Pro also provides the corresponding KoPL program. To simulate the realistic case where KB is incomplete, following (Sun et al., 2019; Shi et al., 2021), we randomly discard 50% triples in the KB and take Wikipedia as supplementary text corpus.
Musique (Trivedi et al., 2022) is a multi-hop QA dataset over text, including 25k 2-4 hop questions. We evaluate our framework under Musique-Ans setting where all the questions are answerable. Its questions are carefully constructed from several single-hop QA datasets via manually composition and paraphrase, and are hard to cheat via reasoning shortcut. For each complex question, Musique gives 20 paragraphs (including annotated evidence paragraphs and distractor paragraphs) as the corpus. Specially, for each question in the training set, Musique also provides a golden atomic representation, together with the answer and the evidence paragraph of each atomic question. In addition to the given paragraphs, we choose Wikidata as the KB to acquire additional knowledge.

heading: 5.2 Implementations
text: KQA Pro For the experiments of KQA Pro, a key challenge is that there are no annotations for atomic representation, which are required for training the question decomposer and generator in RoHT. Because the KoPL program of a complex question follows context free grammar, every atomic question will correspond to a specific span of the program. Therefore we first split the KoPL program into subprograms according to the grammar, then use each sub-program to generate the atomic question by applying BART model fintuned with the (KoPL, question) pairs from the original dataset. For the answers for each atomic question, we execute the corresponding sub-programs on the KB to get corresponding answers. Using these constructed atomic representations, we train two BART-base models as the question decomposer and generator, respectively.
For the scheduler, we directly use the semantic parser trained by (Cao et al., 2022a) on KQAPro, and set the precision threshold γ to be 0.7. We train a RoBERTa-large as the evidence selector via weak supervised method: for each question in the training set and constructed atomic representations, we first use BM25 to recall 10 related paragraphs from wikipedia, then take the paragraphs that contain the answer as positive samples and take other recalled paragraphs as negative samples. For the text executor, we also train a BART-large reading comprehension model on these positive samples.
Musique Since Musique provides golden atomic representation for every complex question in the training set, we directly use them to train BARTbase models as question decomposer and generator. For the scheduler, we adapt semantic parser trained by (Cao et al., 2022a) on Wikidata. The KB precision threshold γ is set to be 0.4, which is determined by the top-10 types of questions with the highest precision. We train the RoBERTa selector model on complex and atomic questions in the training set together, taking annotated evidence paragraphs as positive samples and distractor paragraphs as negative samples. For the text executor, we pre-train a Longformer-large (Beltagy et al., 2020) reading comprehension model on SQUAD (Rajpurkar et al., 2016), then finetune it on complex questions and atomic questions of Musique.

heading: 5.3 Baselines
text: we compare RoHT with several representative methods for complex QA, including memory-based methods, graph-based methods, and XQA methods. KVMemNN (Miller et al., 2016) stores encoded knowledge in key-value memory and iteratively reads the memory to update the query vector to conduct multi-hop reasoning. RGCN (Schlichtkrull et al., 2018) is a variant of graph convolutional network and utilizes the graph structure of KB to tackle complex questions. BART KoPL (Cao et al., 2022a) is a BART-based semantic parser which can convert complex question into KoPL program. It achieves over 90% accuracy on KQA Pro on the complete KB. SA (Trivedi et al., 2022) is a two-stage model that first uses a RoBERTa-large selector to rank and select the K most relevant paragraphs with the question and then uses a Longformer-large answerer to predict answer based on selected paragraphs. EX(SA) (Trivedi et al., 2022) is the state-of-the-art model on Musique. It first explicitly decomposes the complex question into atomic representation and then calling SA model repeatedly to answer each atomic question in order. TransferNet (Shi et al., 2021) iteratively transfer entity scores via activated path on the relation graph that consists of both text-form relations and KB-form relations. It is existing state-of-the-art model that utilizes both KBs and text as knowledge soruces, and nearly solves MetaQA. We reimplement it on both KQA Pro and Musique, and the details are shown in Appendix D. RoHT: RoHTKB, RoHTtext and RoHTmix denote the RoHT models that only use KB, only use text and use both KB and text, respectively.

heading: 5.4 Main Results

heading: 5.4.1 Results on KQA Pro
text: The experimental results for KQA Pro are shown in Table 1. When using only the incomplete KB, RoHTKB model respectively improves EM by 21.22, 4.17 and 0.90 compared to KVMemNN, RGCN and BART KoPL, showing the benefit of integrating the answers of sub-questions of different levels. After adding Wikipedia as supplementary text corpus, RoHTmix yields substantial improvement compared with RoHTKB (7.51 on EM), demonstrating the effectiveness of utilizing knowledge from KB and text together. RoHTmix also outperforms TransferNet, which is end-to-endly trained with a mixed relation graph, by a large margin (29.65 on EM). This is because unlike graphbased methods, RoHT explicitly shows the compositional structure of a complex question in natural language form via HQDT generation, and thus can retrieve answers from the KB and text with more advanced and flexible sub-modules (e.g., semantic parser and reading comprehension model). Moreover, our designed atomic operations in the HQDT also enable RoHT to solve a wide variety of complex questions: we can see that RoHTmix achieves the best results on 6 types of questions among 7 types, showing comprehensive reasoning capacity.

heading: 5.4.2 Results on Musique
text: Table 2 presents the results on the dev set of Musique dataset. As expected, our RoHT models show significant improvement over all the baselines. With only given paragraphs, RoHTtext improves EM/F1 by 13.8/14.3 and 11.6/11.9 compared with SA and EX(SA), respectively; With both text and KB, the performance of RoHTmix is also remarkably better than TransferNet (62.3 v.s. 10.9 on F1). Comparing RoHTtext and RoHTmix,
we can also see some benefits of supplementing the text information with KB information, though the improvement is smaller than supplementing the KB with text on KQA Pro because KBs have lower coverage than text and the semantic parser is not specially finetuned for questions of Musique. We submit the predictions of RoHTmix on the test set and achieve 63.6 F1 score, which significantly outperforms the best public result 52.3.

heading: 5.5 Further Analysis

heading: 5.5.1 Effect of Scheduler
text: To show the effect of the scheduler module, we remove it from the RoHTmix model, i.e, default that the KB and recalled/given text paragraphs are suitable for all questions in the HQDT, and evaluate the performance again on the dev set of KQA Pro and Musique. The results are shown in Table 3. We can see that after discarding the scheduler, the EM performance on KQA Pro and Musique drops by 5.8 and 7.4, respectively. Therefore, it is important to use the scheduler to select suitable knowledge sources for each question.

heading: 5.5.2 Effect of Hierarchical Decomposition
text: Many existing methods generate non-hierarchical decomposition of complex questions, similar to the atomic representation, to assist reasoning (Min et al., 2019; Wolfson et al., 2020; Deng et al., 2022). To demonstrate the superiority of hierarchical decomposition, we compare our RoHTmix model with
RoATmix model, which uses the same scheduler, executors, and aggregator as RoHTmix, but solves the complex question by directly answering the atomic questions in its atomic representation in order. As shown in Table 3, RoHTmix outperforms RoATmix by a large margin on both KQA Pro and Musique. This is because the hierarchical structure of HQDT enables RoHT model to fuse the knowledge from KBs and text at different question levels, and to discard wrong answers via comparing the problisitic scores of answers.
To further understand the reason, we show a case from Musique in Figure 3. We can see that both RoHTmix and RoATmix fail to answer the question “Where did (Titian) die?” (q4 in the left, a02 in the right). However, RoHTmix directly extracts the correct answer of q1 from text and finally gets the correct answer of q0 with the highest score, while RoHTmix fails to solve a03 because it must rely on the wrong answer from a02.

heading: 6 Conclusion
text: In this paper, we propose RoHT, an understandingreasoning XQA framework that uses both a KB and a text corpus to derive answers of complex questions. RoHT first builds the HQDT for a complex question to understand its hierarchical compositional structure, then conducts recursive probabilistic reasoning over the HQDT to solve the question, integrating answers from the KB, text, and sub-questions. Experiments show that RoHT significantly outperforms previous methods. We also demonstrate the superiority of HQDT compared with non-hierarchical decomposition.

heading: 7 Limitation
text: Currently, RoHT framework is restricted to incorporating KBs and text. However, since RoHT retrieves answers from each knowledge source in a separate way, it could in principle utilize knowledge from more heterogeneous sources such as tables, and we will study this in future work. In addition, a device with large storage space and memory is needed for the storage and usage of Wikipeida and Wikidata.

heading: 8 Ethics Statement
text: The data used in this paper are drawn from publicly published datasets, encyclopedias and knowledge bases. Most of them do not involve sensitive data.

heading: 9 Acknowledgement
text: This work is supported by grants from the Institute for Guo Qiang, Tsinghua University (2019GQB0003) and Cloud BU, Huawei Technologies.

heading: A Atomic Operations
text: We design 6 atomic operations: Verify, SelectBetween, SelectAmong, Count, Intersection, Union, to support various reasoning capacity. We show their input, output, and examples in Table 4.

heading: B Get Atomic Representation from Leaf Nodes
text: Algorithm 1 describes that how to get the atomic representation of a question qi ∈ T from the leaf nodes in the sub-tree rooted by qi.
Algorithm 1 Get Atomic Representation from Leaf Nodes Input: An HQDT T and a index i. Output: qi.atoms
1: function DFS(j, atoms, ids, n) 2: if qj is a leaf question then 3: n← n+ 1 4: ids[j]← n 5: a← qj 6: for k in GetRefTokens(qj) do 7: if qk is a leaf question then 8: a← ModifyRefTokens(a, k, ids[k]) 9: else
10: a← ModifyRefTokens(a, k, ids[edk]) 11: atoms.append(a) 12: return 13: for k ← stj , . . . , edj do 14: Dfs(k) 15: 16: qi.atoms← [] 17: ids← empty dict 18: Dfs(i, qi.atoms, ids, 0)

heading: C Pseudocode for Building HQDT
text: Algorithm 2 shows the pseudocode for generating the HQDT of a complex question with probability.

heading: D Reimplementation of TransferNet
text: To reimplemente TransferNet, we build the mixed relation graphs that consist of both label-form relations (i.e., KB triples) and text-form relations for KQA Pro and Musique, respectively, and train the models with the open source code. We show the details of graph building as follows.
KQA Pro We follow the method used by the original paper on MetaQA to build the relation graph of KQA Pro. As mentioned in Section 5.2, we use half of its KB triples as the label form. We constructe the text form by extracting sentences from Wikipedia. Following the original paper, we use exact match of surface forms for entity recognition
Algorithm 2 Generation of HQDT Input: a complex question q0, a question decomposer Mθ , a question generator Mϕ. Output: a list T representing the HQDT, where element (qi, pig, fai) in T denote a sub-question qi, certainty score of qi and the father of qi, respectively.
1: function REARRANGEREFTOKENS(ar) 2: atoms← [] 3: ids← empty dict 4: h← 0 5: for (i, ai) in ar do 6: h← h+ 1 7: ids[i]← h 8: for k in GetRefTokens(ai) do 9: ai ← ModifyRefTokens(ai, k, ids[k])
10: atoms.append(ai) 11: return atoms 12: 13: ([a01, . . . , a0n0 ], ld)←Mθ(q0) 14: n← n0 15: T ← [] 16: for i← 1, 2, . . . , n0 do 17: (qi, pig)← (a0i , ld) 18: ari ← [(i, a0i )] 19: if a0i contains referring tokens then 20: r1, . . . , rh ← GetRefTokens(a0i ) 21: n← n+ 1 22: arn ← [] 23: for j ← r1, . . . , rh, i do 24: if faj has been identified then 25: qi ← ModifyRefTokens(qi, j, faj) 26: j ← faj 27: faj ← n 28: T .append((qj , pjg, faj)) 29: arn.extend(arj) 30: qn.atoms← RearrangeRefTokens(arn) 31: Ln = Serialize(qn.atoms) 32: (qn, lng )←Mϕ(Ln) 33: png ← ld · lng 34: T .append((q0, 1, 0)) ▷ directly use q0 as root 35: T ← ReIndexByBFS(T ) 36: return T
and linking. For every entity in the KB, we recall all the paragraphs in Wikipedia titled by it, then take the entity as subject and other relevant entities appeared in these paragraphs as objects. The sentences that contain the objects are selected as the relation texts. The recall of answer is 51%, i.e, for 51% questions, there exist a complete path from the topic entity to the answer in the relation graph, and this is a upper bound for the performance of TransferNet.
Musique For each question in Musique, we utilize the 20 given paragraphs to build individual relation graph. Specifically, we first identify entities mentioned in these paragraphs via Spacy (Honnibal et al., 2020) and exact match of surface forms with Wikidata entities. Then we take the co-occuring
sentences of two entities as the text-form, and take the triples in Wikidata whose subject or object is one of these entities as the label-form. The recall of answer is 72%.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
3 A2. Did you discuss any potential risks of your work? Section 8
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4
3 B1. Did you cite the creators of artifacts you used? Section 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We follow the same license
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data used in this work are drawn from publicly published encyclopedias, knowledge bases and datasets.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 5
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: However, since RoHT retrieves answers from each knowledge source in a separate way, it could in principle utilize knowledge from more heterogeneous sources such as tables, and we will study this in future work. In addition, a device with large storage space and memory is needed for the storage and usage of Wikipeida and Wikidata