HPE: Answering Complex Questions over Text by Hybrid Question Parsing and Execution
Authors: Ye Liu
Semih Yavuz
Rui Meng
Dragomir Radev
Caiming Xiong
Shafiq Joty
Yingbo Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: Multi-hop Question answering; Neuro-Symbolic method; Tree Structure Reasoning; Semantic Parsing

Abstract: The dominant paradigm of textual question answering systems is based on end-to-end neural networks, which excels at answering natural language questions but falls short on complex ones. This stands in contrast to the broad adaptation of semantic parsing approaches over structured data sources (e.g., relational database, knowledge graphs), that convert natural language questions to logical forms and execute them with query engines. Towards combining the strengths of neural and symbolic methods, we propose a framework of question parsing and execution on textual QA. It comprises two central pillars: (1) We parse the question of varying complexity into an intermediate representation, named H-expression, which is composed of simple questions as the primitives and symbolic operations representing the relationships among them; (2) To execute the resulting H-expressions, we design a hybrid executor, which integrates the deterministic rules to translate the symbolic operations with a drop-in neural reader network to answer each decomposed simple question. Hence, the proposed framework can be viewed as a top-down question parsing followed by a bottom-up answer backtracking. The resulting H-expressions closely guide the execution process, offering higher precision besides better interpretability while still preserving the advantages of the neural readers for resolving its primitive elements. Our extensive experiments on MuSiQue, 2WikiQA, HotpotQA, and NQ show that the proposed parsing and hybrid execution framework outperforms existing approaches in supervised, few-shot, and zero-shot settings, while also effectively exposing its underlying reasoning process.

HPE: Answering Complex Questions over Text by Hybrid Question Parsing and Execution
Ye Liu', Semih Yavuz' Rui Meng', Dragomir Radev Caiming Xiong'. Shafiq Joty', Yingbo Zhou' Salesforce Research," Yale University yeliuesalesforce com
Abstract
The dominant paradigm textual question answering with end-to-end neural models ex- cels at answering simple questions but falls short on explainability and dealing with more complex questions. This stands in contrast to the broad adaptation semantic parsing ap- proaches structured data sources (e-g., re- lational database), that convert questions to log- ical forms and execute them with query engines. Towards the goal of combining the strengths of neural and symbolic methods, we propose framework of question parsing and execution for textual QA. It comprises two central pil- lars: (1) parsing question of varying complex- ity into an intermediate representation, named H-expression, which symbolically represents how an answer to the question can reached by hierarchically combining answers from the primitive simple questions; (2) to execute the re- sulting expression, we design hybrid executor, which integrates deterministic rules to translate the symbolic operations with a drop-in neural reader answer each simple question. The pro- posed framework can be viewed as top-down question parsing followed by a bottom-up up an- swer backtracking. H-expressions closely guide the execution process, offering higher precision besides better interpretability while still pre- serving the advantages of the neural readers for resolving primitive clements. Our extensive ex- periments on four different QA datasets show that the proposed framework outperforms ex- isting approaches in supervised, few-shot, and zero-shot settings, while also effectively expos- ing the underlying reasoning !
When the last time Duane Courtney's team beat the winner the 1894-95 Cup?
Question:
QM When
UNIN
[A] What a member of sport
CAM Who is winner of
team of Dusn Courtney?) Tree structure Hexxresssion
1894-95 FA Cup?]
Figure 1: An illustration of H H-expression.
ing to their flexibility and solid performance. How- ever, they often suffer from lack of interpretabil- ity and generalizability to more complex scenar- ios. Symbolic reasoning, on the other hand, relies on producing intermediate explicit representations such as logical forms or programs, which can then be executed against a structured knowledge base (e.g., relational database, knowledge graph, etc.) to answer questions (Gu et al., 2022; Baig et al., 2022; Zhao et al., 2023; Wang et al., 2023). These methods naturally offer better interpretability and precision thanks to the intermediate symbolic repre- sentations and their deterministic executions. How- ever, they might be limited in expressing a broad range of natural questions in the wild depending on the semantic coverage of the underlying symbolic language and its grammar.
Neural Module Networks (Andreas et al., 2016; Gupta et al., 2019; Khot et al., 2021) 1) have been pro- posed to combine neural and symbolic modality to- gether. However, they require a symbolic language and corresponding model that only covers limited scenarios in the specific task domain. To apply this approach new tasks or domains, new lan- guages and neural modules need be introduced. Therefore, designing a generalizable framework that uses high-coverage symbolic expression and a flexible neural network that can be versatilely used in various scenarios becomes our goal.
1 Introduction
mnd-to-end neural models that transductively learn map questions to their answers have been the dominating paradigm for textual question answer- ing (Raffel al., 2020; Yasunaga al., 2021) ow-
In this work, we propose a Hybrid question Parser and Execution framework, named HPE, for textual question answering, which combines neural and symbolic reasoning. We introduce H-
'Our code is available at https:/ www.. com/ aa salesforce/Hpp

---

expression as an explicit representation of the orig- inal question, which contains primitives and op- erations (Liu et al.. 2022). As shown in Figure I, we consider the single-hop questions as primi- tives (leaves) and use symbolic operations (internal nodes) to connect them hierarchically. We intro- duce a semantic parser based on a seq2seq frame- work to parse a complex question into its corre- sponding H-expression in top-down manner.
modules by performing text span extraction and arithmetic operations. Khot et al. (2021) propose an NMN variant that decomposes a question into a sequence of simpler ones answerable by different task-specific models. Systematic question decom- position has also been explored in (Talmor and Be- rant, 2018 8; Min et al., 2019; Wolfson et al., 2020; Zhang et al., 2023).
Although our framework shares some similar- ities with this line of studies, there crucial difference in that we keep both symbolic and neu- ral representations coincide, whereas they use a neural model to replace the non-differentiable sym- bolic representation to be able to train end-to-end. Furthermore, our main contribution is the use of H- expressions and the answer backtracking procedure for QA and not the question decomposition.
To execute an H-expression, we design hy- brid executor (H-executor), which utilizes a neural model (reader) to answer each single-hop ques- tion, and then uses deterministic rules to hierar- chically compose final answer from the single- hop answers in bottom-up fashion. Notably, H- expression facilitates modularity in that the reader is replacable and its training can be done globally with the massive single-hop QA data.
Our contributions can be summarized as follows:
2.2 Shain-of-Thoughe Reasoning
Architecture We propose to combine the ad- vantages of both symbolic and neural reasoning paradigms by parsing questions into hybrid in- termediate expressions that can be hierarchically executed against the text to produce the final an- swer. Our experiments on MuSiQue (Trivedi et al., 2022b) and 2WikiMultiHoPAA (Min et al., 2019) show that the proposed approach achieves state-of- the-art performance.
series of prior studies focus on generating expla- nations, which can be viewed as reasoning chains. The methods proposed in (Yavuz et al., 2022; Latcinnik and Berant, 2020; Jhamtani and Clark, 2020) formulate the multi-hop QA single se- quence generation, which contains answer along with its reasoning path. Along this line, Large Lan- guage Models (LLMs) have recently shown its ca- pability to answer complex questions by producing step-by-step reasoning chains, known as chains-of- thought (CoT), when prompted with few exam- ples (Wang et al., 2022; Zhou et al.. 2022: Wei et al.. 2022; Lyu et al., 2023) or even without any exam- ple (Kojima et al., 2022; Kadavath et al., 2022).
Generalizability End-to-end neural models are data hungry and may sigmificantly suffer from poor generalization to unseen data, especially in limited resource scenarios. Our design, on the other hand, naturally splits the reasoning process into parsing and execution, through which it intends to disentan- gle learning parse complex questions structurally from learning to resolve simple questions therein, making the process modular. Our few-shot exper- iments on MuSiQue and 2 2WikiMultiHopQA and zero-shot experiments on HotpotQA(Yann et al., 2018) and NQ (Kwiatkowski et al., 2019) suggest that even with less training data, our approach can generalize better to unseen domains.
Even though the generated reasoning path by these methods may provide some explanation on how the question being solved, there no guar- antee that the answer is indeed generated by the predicted reasoning path. Furthermore, CoT reason- ing paths are aimed at providing more context for the LLMs to be able to locate the right answer. This is different from our objective where our symbolic representation gives a deterministic and hierarchi- cal reasoning (execution) path derive an answer.
Interpretability The execution process of our model the same as its reasoning process. Trans- parency of our approach facilitates spotting and fixing erroneous cases.
3 Approach
We formulate textual question answering as the task of answering a question g given the textual evi- dence provided by a set of passages. We assume ac- cess to a dataset of tuples {(i dj P where a; is a text string that defines the correct answer to question q with P; being the passage set.
2 Related Work
2.1 Neural Symbolic Systems
Gupta et al. (2019) introduce a neural module net- work (NMN), which solves QA using different

---

Question:
example. Specifically, the JOIN[q2(Ans#I), qll op- cration used for linear-chain reasoning type qI is an independent question that can be answered directly, while q2's s execution depends on qI's an- swer. In the execution step, the operations will be executed in a sequence: first ql, then the answer of q1 will be used to replace the placeholder (Ans#1) in q2. The AND [q2. q1] and UNION lq2, ql] oper- ations respectively return the common answers all of the answers for q2 and ql; see Figure 2 for an example.
When was the last time Duane Courtney's team beat the winner of the 1894-95 FA Cup?
H-Parser
H-Expression:
JOIN QA! When was the last time Ans#2 beat AnsM1? , UNION ( QAI What member sports team of Duan Courtney )
QA( Who winner 1894-95 Cup)]] )]
JOIN
QA( When was the last time Ansf2 beat Ans#1?
UNION
The COMP 1q2, ql] operation is used to deter- mine if the answers of q2 and ql are equal and it returns "Yes" or "No". The COMP_ [q2.ql] and COMP_> [q2. q1] operations respectively select the question entity corresponding to the smaller or the bigger answer of q2 and ql. Finally, the SUB and ADD are numeric operations and perform subtrac- tion and addition, respectively.
Q1
QAI What is member of sports team of Duan Courtney?
QA Who
1894-95 FA Cup?)
Figure 2: An overview of parsing with H-parser, which involves translating the input question into H- expression, and subsequently reshaping it into a tree- structure, facilitating the determination of the node's execution sequence.
In this work, we cast the task question pars- ing with hybrid execution. For a given question Q. a question parser (H-parser) is tasked to generate the corresponding H-expression 1 (Sec. 3.1). This along with the supporting passage set P; are then given to the execution model (H-executor) to gen- crate answer (Sec. 3.2). The H-executor uses reader for simple question answering and exe- cutes the symbolic operations based on answers in a bottom-up manner to get the final prediction.
H-expression Generation The semantic pars- ing process of queries over knowledge bases or databases typically needs to consider the back- ground context to match natural questions to their corresponding logical forms with the specified schema, which is a necessary condition to execute in knowledge base (Ye et al., 2022) or table (Lin et al., 2020). However, in textual QA, the ques- tion parsing process should be context-independent as we want the meaning of the original question and the H-expression to be equivalent without any additional information from the context.
3.1 H-parser
To improve the compositional generalization, we follow (Liu et al., 2022) to define H --expression as a composition of primitives and operations.
Our parser is a Seq2Seq model that takes a natural question q as input and generates its H- expression I as output. Seq2seq formulations have been successfully used for parsing tasks (Vinyals et al., 2015). We use T5 model (Raffel et al., 2020) as the basis of our parser, as it demonstrates strong performance on various text generation tasks. We train the model by teacher forcing - the target H- expression is generated token by token, and the model is optimized using cross-entropy loss. At in- ference time, we use beam search to decode the top- l target H-expression in auto-regressive manner. It is easy to transform the nested H-expression binary tree structure in a top-to-down manner, where the primitives constitute the leaf nodes and the internal nodes represent the deterministic sym- bolic operations.
H-expression Grammar We consider single- hop questions as primitives, which the atomic units that form complex question. We use opera- tions represent relations between primitives. We consider eight types of operations: JOIN, UNION, AND, COMP_= COMP_>, COMP_ <, SUB, and ADD. Each is a binary operation that takes two operands as input, written as OP [42, q1], where each of q and q2 can be a single-hop question or result of another operation. In the exccution step, ql is executed first, then q2. The operations can be hi- erarchically combined into more complex one, for example, JOIN [q3, UNION [q2, q1]]. For single-
hop question, its H-expression is the question itself. We describe the operations in Table with an

---

Operation & Return type
Description & Example
JOIN] q2(Anc+1), ql] Text span
92's s execution depends on ql answer to replace the placeholder "Ansa]" Finally returns q2's answer
Question: Where was birth place film The Man director?
II :eeexpresioo: JOIN| Where Amsel's 'ss place of birth?. Who director of The Iron Man?
Return: New York
UNIONI q2. qu)
Execuiles 42 and ql simulannously and retums a dictionary as [Antt al, Amsi2 a2), where al/2 the unswer of q1/2.
Dictionary
Question: Which
Horndean located
what McDonaldization mateed after?
H-expression: UNIUN] Which state is Horndean located in!. What MeDonaldizaion named after?
Return: [AnsH] McDonlld'ss x. 2
ANDI q2, q1l
Exocuntes q2 and ql simultancously and retums intersection of q2 and ql's answers.
Text spans
Question: Which former member of the Piithurgh Pirates nicknamed "The Cobra"?
II expression: AND Who is the former member of the Fittsburgh Pirates?, Who was nicknamed "The Cobra"??
Return: Dave Parker
COMP_=I
Compure il suswers of 42 and q1 are equal.
Yes/No
Question: Are North Marion High School (Oregon) and Scoul High School both located in the same country?
H-expression: COMP =| Which is country of North Marion High School (Oregon)?. Which is country of Scoul High School? ]
Return: No
COMP_<192.0 qll
Compare the answers of q2 and ql and return
catily q2 or gl.
COMP 109_>192211111
Question: Which film was came out first, Blind Shaft The Mask of Fu Manchu?
Main Entity in q21g]
II expression: COMP of When is publication date of Blind Shaft?, When is publication date of The Mask of Fu Manchu?
Return: Mask Fu Manchu
SUR| 92.q1 qi
Subtract q2's nameric answer from ql'
Number
Question: How many years does Giuseppe Cesari live?
H-cxpression: SUBI When does Giuseppe Cesari dead?, When does Ciuseppe Cesari bora?
Return 72
ADD 922 Number
Add q2's and ql's numerie answers.
Question: How many siblings does Mary Shelley have?
H-cxpression: ADD How many sisters does Mary Shelley have?, How many brothers does Mary Shelley have?
Return:
Table I: Operations defined in our H-expressions and its corresponding example of question, H-expression and return; q2 and ql are single-hop natural questions.
Question:
and the left branch recursively. This is similar to in- order traversal (with opposite leaf order). As shown in Figure 3, the question "who is winner of 1894-95 FA Cup" is the first primitive be exccuted and the single-hop reader is called to answer it, which yields the answer "Aston Villa". Then internal node operation UNION is visited, which stores "Aston Villa" as Ans#I. Then the left primitive "what is member of sports team of Duane Courtney" is vis- ited, and the single-hop reader is called again to answer this. The reader yields "Birminghan City", which is stored as Ams#2. Operation JOIN is vis- ited next, which replaces the placeholder Ans#1 and Ans#2 with the stored answer and produce a new primitive "When was the last time Birminghan City beat Aston Villa". This resulting primitive is answered by the single-hop reader, which predicts the final answer "I December 2010".
When was the last time Duane Courtney's team best the winner of the 1894-95 Cup?
QAI When was the last time Birminghan City"
Answer:
ION
Deoember 2010
Aston Villa?)
CAN Winten the lest time Anstt best Anshi71
UNION
Plug-in Single-hop Reader
Ans
AnsR2: Birmingban City
Ansel:
CAI What member of sports learn Duana Coustrey?)
OA Who winnerdf
An overview of the H-executor. It involves execution of the question node and utilization of a plug- in neural reader to provide answer feedback for each question. Subsequently, the deterministic symbolic inter- preter executes the expression to yield the final answer.
Figure 3:
3.2 H-executor
Unlike the execution in databases or knowledge bases which is fully program-based, our execution has both a neural component and symbolic com- ponent. All the primitives in the H-expression tree are executed by a neural reader and the results are aggregated by the symbolic rules expressed by the operations in the internal nodes.
Plug-in Reader Network The reader network serves as plug-in fasion that can be replaced with any model. In our experiment, we commence with generative encoder-decoder model, FiD (Izacard and Grave, 2021) and further extend it by exper- imenting with various versions of FiD, each pos- sessing distinct input and output configurations. Each supporting passage is concatenated with the question, and processed independently from other
Specifically, the H-executor traverses from the rightmost primitive, followed by its parent node

---

passages by the encoder, which makes it efficient. The decoder attends to the concatenation of all en- coded representations from the encoder. To distin- guish different components, we add the special to- kens question on: title: and context: be- fore the question, title and text of each passage. Note that the reader network is detachable and may be replaced by any generative or extractive reader model our choice.
20 supporting passages. It contains 19,938, 2,417, and 2,459 instances for train, dev and test sets, respectively, with 2hop1 (meaning questions with 2 hops and entity), 3hopl, 4hopl, 3hop2, 4hop2 and 4hop3 reasoning types.
2Wikimulti-hopQA (2WikiQA) (Ho et al., 2020) requires models to read and perform multi-hop reasoning over 10 passages. Three types of reasoning are included, namely compar- ison, bridge, and bridge-comparisee 167,4 ,44, 12,576, and 12,576 for train, dev and test, respectively.
Our assumption that the single-hop questions are easier to answer and it feasible to have a global single-hop reader, which can be adapted to an unseen data. To reduce training cost, we first train a T5-large in the reading comprehension set- ting, with one positive passage. For this, we lever- age the large-scale QA pairs from Probably-Asked Questions/PAQ (Lewis et al., 2021). Then we the trained T5-large model to initialize the FiD model and further train the model using training sets of TriviaQA (Joshi et al., 2017), SQuAD (Ra- jpurkar et al., 2016), BoolQ (Clark et al., 2019) in a multiple passage setting (with one positive pas- sage and nineteen negative passages). We believe, our model, being trained on multiple datasets, can be used to unseen questions in zero-shot manner. It can also boost the performance when used to initialize in fine-tuning setting.
It contains
For each complex question, MuSiQue provides reasoning type and the decomposed single ques- tions with their answers. We use JOIN operation combine linear-chain type questions together and use AND to combine intersection type questions. In 2WikiQA, we use evidences (in form of triplet <subject, relation, object>) and reasoning type to create the H-expression. We first convert the subject and relation into natural questions using templates and the object is the answer of this natural question. the operation to combine those single- hop questions into an H-expression based on their reasoning type. In Table 1, we show few exam- ples of complex questions and their corresponding H H-expressions; see Appendix for more examples.
Then,
4 Experiments
Evaluation Metrics. We use official evaluation scripts for each dataset with two metrics to measure answer exact match accuracy (EM) and answer token-level accuracy (F1).
We conduct experiments on two multi-hop multi- passage textual QA datasets, MuSiQue and 2Wikimulti-hopQA. which contain complex ques- tions and the corresponding decomposed simple questions for the supervised setting. We also test models' generalization on the few shot setting us- ing 5-20% of the training data. In real scenarios, neither the decomposed questions nor the complex- ity of questions is known. Therefore, we also in- vestigate our models under the zero-shot setting on both complex (HotpoiQA) and simple (NQ) QA datasets. In the end, we carry out a case study to show the interpretability of our framework.
4.1.2 Baselines
Press et al. (2022) and Trivedi et al. (2022a) make use of large language models like GPT-3 (Brown et al., 2020). They iteratively generate an answer- able question, use retrieval to get supporting pas- sages, and answer the question based the trieved passages. SA (Trivedi et al., 2022b) is the state-of-the-a model on the MuSiQue dataset, which first uses a RoBERTa based (Liu et al.. 2019) ranking model to rank supporting passages and then uses an End2End reader model to answer com- plex questions using the top-k ranked passages. EX(SA) (Trivedi et al., 2022b) decomposes a com- plex question into single-hop questions and builds directed acyelic graph (DAG) for each single- hop reader (SA) to memorize the answer flow. NA- Reviewer (Fu et al., 2022) proposes a reviewer model that can fix the error prediction from incor- rect evidence. We include FiD (Izacard and Grave,
4.1 Supervised Experiments
4.1.1 Datasets
We describe each dataset and then explain how to convert the original data into the training format for both question parsing and execution.
MuSiQue (Trivedi et al., 2022b) contains multi- hop reasoning questions with different number of hops and question entities which can be asked from

---

MuSiQue
2WikiQA
EM
F1
EM
F1
Self-ask Search (Press al., 2022)
15.2
40.1
Large LM
IRCoT (Trivedi et al.. 2022a)
35.5
65.2
SA (Trivedi et al., 2022b)
52.3
79.5
SOTA
EX(SA) (Trivedi et al., 2022b)
49.0
71.2
NA-Reviewer (Fu et al., 2022)
76.9
82.3
FiD
37.6
45.3
76.9
80.8
FiD IPT
40.0
48.8
78.8
83.0
End2End
F.D Ans Ans
36.1
44.8
76.9
80.5
FDCQ->1.F4Am
33.7
42.1
74.2
77.6
HPE
42.9
50.1
80.1
84.5
Ours
HPE
45.5
53.7
84.7
87.7
Table 2: Answer Exact match (EM) and Fl scores dev/test split of MuSiQue and 2WikiQA. PT represents pre-training on reader network. The methods in Large LM and SOTA are reported from the previous work. The methods in End2End is implemented by us following the training details in the paper.
2021) as the baseline End2End reader model. In the original FiD, it takes the question as well as the sup- porting passages input, and generates the answer sequence of tokens. Morcover, we propose two variants of FiD to compare the influence using H- expression: one uses H-expressions as the input, instead of original questions, to generate answers (referred to as FiDIF->Annl and the other uses questions as input to generate both H-expressions and answers (referred to FiDcq->LF+Aus)
IRCoT, which are based on large language mod- cls and search engines, underperform most super- vised models. This indicates that multi-hop multi- paragraph question answering is difficult task, and there still has an evident gap between super- vised small models and large models with few-shot or zero-shot.
Moreover, our framework outperforms the pre- vious SOTA methods on both datasets. We notice that the baseline EX(SA) underperforms SA by a large margin, but our HPE outperforms FiD by 5.3% on MuSiQue EMM This shows the difficulty to build good H-cxpression and executor. More- over, EX(SA) gets a bad performance on 2Wik- iQA, which shows that using DAG to represent the logical relationship between sub-questions is not adaptable any reasoning type. Compared with the End2End baseline (FiD) that our model is built on, our framework with an explicit representation performs much better.
4.1.3 Implementation Details
We describe fine-tuning details for question parsing and single-hop reader models in Appendix B.
Pre-training (PT) To pretrain the single-hop reader, we use a subset of PAQ (Lewis et al., 2021) consisting of 20M pairs, which is generated based named entities and the greedy decoded top-1 se- quence with the beam size of 4. We train a TS-large model for 400k steps, with one gold passage, maxi- mum length of 256 and batch size of 64. Then initialize FiD with the PAQ pre-trained model and further train for 40k steps, with batch size and 20 supporting passages, on combined train- ing sets of TriviaQA (Joshi et al., 2017), SQuAD (Rajpurkar et al., 2016) and BoolQ (Clark et al., 2019). Our code is based on Huggingface Trans- formers (Wolf et al., 2019). All the experiments are conducted on cloud instance with eight NVIDIA A100 GPUs (40GB).
As for FiDLF->Ans and DLDQ->LF+Aus using H-expression as the input or output of the Seq2Seq model, expecting this facilitates the model cap- ture the decomposition and reasoning path in an implicit way, does not help the model. This sug- gests that only the proposed execution method can help the model capture the logical reasoning repre- sented in the H-expression.
4.2 Few-shot Results
To illustrate the generalization ability of our frame- work, we show the analysis of our method under the few-shot setting in Table 3. We run three ex- periments, random sampling 5, 10, and 20 percent-
4.1.4 Fine-tuning Results
We present our main results on MuSiQue and 2Wik- iQA in Table 2. We observe that Self-ask and

---

Method
MuSiQue
2WikiQA
HotpotQA
EM
F1
EM
F1
EM
F1
5%
FiD
29.7
38.7
55.6
60.6
Standard (Yao al., 2022)
28.7
HPEFiD
34.6
42.3
67.0
72.3
CoT (Wei al., 2022)
29.4
FiD (PT)
32.5
44.7
10%
FilD
31.2
40.5
58.3
63.0
НPE(PT)
32.6
43.4
HPEFiD
35.2
43.3
68.3
73.8
Umon HPE+FID Supervised SOTA
47.2
55..I"
20%
FiD
31.9
41.0
69.1
73.5
72.3
84.9
HPEFID
36.4
44.2
73.0
78.0
Table 4: Zero-shot performance on HotpoiQA. Standard and CoT are prompted method using large language model like GPT3 (Brown et al., 2020).
Table 3:
Few-shot setting Exact match (EM) and Fl scores on test/dev split of the MuSiQue and 2WikiQA.
age of the training data. We use the End2End FiD model the baseline, which inputs complex ques- tions and generates the answers as token sequences, In 5% of MuSiQue dataset, it shows that our frame- work obtains a 4.9% absolute gain on MuSiQue EM score in comparison to the FiD model. More- over, with 20% MuSiQue training data, our frame- work achieves 36.4 EM, which is a comparable performance with FiD trained on full-data (37.6 EM). Similar trends are also observed 2Wik- iQA. In summary, the overall experiment shows that our model has better generalization ability than the End2End model, which obtained by decom- posing complex questions into single-hop ones and representing in H-expressions.
single-hop questions.
4.3.2 Global Question Parser
To seamlessly generate H-expressions unseen questions, we need a global question parser. This question parser can understand the complexity of the question, which means can decompose a com- plex question into several simple questions and keep the simple question as is. To get a global question parser, we train pretrained generative model T5 (Raffel et al., 2020) to convert questions to H-expressions using MuSiQue and 2Wikimulti- hopQA datasets. As the two datasets are not the same size, categorize the complex question based on their reasoning type and sample the same amount of data for each category. To endow the model with the ability of understanding question complexity, we also use the simple questions in those datasets (the H-expression of a simple ques- tion is itself). Moreover, we decouple the composi- tion of complex H-expressions into few of simple H-expressions to ensure the coverage of all levels of complexity.
4.3 Zero-shot Results
We expect the H-parser to work well on questions of varying levels of complexity. To verify this, we test the models on two benchmarks HotpotQA and Natural Questions without any tuning. The former does not contain any decomposed questions, and the latter contains common real-world questions.
4.3.3 Zero-Shot Results on HotpotQA
4.3.1 Dataset
We show the HotpotQA results in Table 4. We use FiD pre-trained on PAQ and TriviaQA, SQuAD and BoolQ as zero-shot reader. Our framework outperforms both Standard and CoT, using prompt- based large language models. This shows that with the hybrid question parsing and execution frame- work, a small language model is generalizable on unseen questions. Compared with FiD (PT), we get comparable performance. But checking the union of HPE and FiD, which takes the correct predictions from both methods, find 15% ab- solute gain can be obtained. This shows that HPE correctly answers around 15% of questions that FiD predicts incorrectly, with the help of the ques- tion decomposition and symbolic operation. On the other hand, we conjecture that the reason that HPE
HotpotQA we use the distractor setting (Yang al., 2018) that a model needs answer each ques- tion given 10 passages. To produce correct answer for a question, the dataset requires the model to reason across two passages. Note that two main rea- soning types bridge and comparison in HotpotQA are included in MuSiQue and 2WikiQA.
Natural Questions (NQ) (Kwiatkowski et al., 2019) contains open-domain questions collected from Google search queries. Usually, NQ is treated simple question dataset and previous works usually use End2End multi-passage reader like FiD, However, we argue that certain questions in NQ involve multi-hop reasoning and the model perfor- mance can be improved by decomposing them into

---

wrongly predicts some questions that the global question parser fails to generate H-expression cor- rectly. Hence, worth exploring how to design a generalizable global question parser in future work.
Executability of H-parser We employ exe- cutability as a metric to assess the quality of the generated outputs. Table we showcase the rate at which the T5-large model generates executable logical forms within the Top-k decoded list. As we can see, the Top-1 logical forms demonstratc a high executable rate (96% 98%). underscoring the model's 's expertise in producing syntactically correct generations. As the beam size broadens, majority of the Top-k logical forms prove to be executable, greatly enhancing our inference process.
4.3.4 Results on NQ
We use the global question parser to decompose NQ question in zero-shot manner. If a question is recognized as single-hop reasoning and cannot be further decomposed, the parser will keep the ques- tion unchanged. We use the DPR model (Karpukhin et al., 2020) retrieve the Top-20 documents from Wikipedia as the supporting documents. Among the 8k dev set examples, 32 questions have been decomposed into single-hop questions with the log- ical operations and the rest are left as is. For ex- ample, a question "when did the last survivor of the titanic die" is converted into the H-expression "JOIN !when did Al die, who the last person survive the titanic]". The result in Table shows that HPE can handle questions of different com- plexity levels and will not degenerate on simple questions.
MuSiQue
2WikiQA
Top-1 Top-10
95.6 99.7
98.3 100.0
Table 7: The execution rate of the H-parser.
Impact of H-executor Our hybrid executor is combined with symbolic operations and replace- able reader network. We analyze the influence of different reader networks to the final performance and experiment with different versions of FiD. Sup- portFiD generates both answers and the supporting document titles. SelectFiD is a two-step method that first uses RoBERTa-based (Liu et al., 2019) ranking model to predict the Top-5 relevant doc- uments and feeds them into FiD generate the answer. From results in Table 8, we can see that a better single-hop reader produces better perfor- mance on MuSiQue. The improvement on single- hop reader translates to significant performance boost on complex questions.
EM
F1
FiD (FT) HPE (FT)
51.4
56.2
51.7
56.3
Table 5: Answer exact match (EM) and Fl scores on
dev split of the simple QA NQ.
4.4 Ablation Study
Impact of H-parser We show the performance of different H-parsers. Table 6 shows using T5- large rather than T5-base, can get around 4 percent performance improvement on both datasets. Compared to the result using gold H- expression, there is more room for improvement on the MuSiQue dataset. This might also be the case as the questions in MuSiQue are generally more complex than 2WikiQA.
4.5 Case Study
In this section, we analyze the error cases. More- over, we show the performance under each reason- ing type on MuSiQue and 2WikiQA in Appendix C. In the end, show a of how model reasons on complex question in Appendix D.
Error Analysis There are two types errors of our model prediction. One is the error from the semantic parsing of the H-expression. The other is the eтror from the single-hop question answer on H-execution stage. The percentage of the first type of error is 67% and the second type is 33% on the MuSiQue dataset.
MuSiQue
2WikiQA
EM
F1
EM
F1
Gold
50.2
57.5
83.6
86.5
T5-base
39.8
46.5
78.9
82.3
Ta-large
42.9
50.1
80.1
84.5
Table 6: Different question parsers and the gold H- expression impact on Answer EM and on MuSiQue and 2WikiQA under same FiD as the single-hop reader.
Even though our model boasts high execution rate, as highlighted in the table, it is not without challenges. In particular, the H-expression gener- ated by our model might not correspond with the

---

FiD(TS-large) T--aas)
FiD(PT) Ans EM/FI
SupportFiD(PT)
SelectFiD(PP)
Ans EM/FI
Ans EM/F1
Ans EM/FI
SP EM/F1
Ans EM/FI
SP EM/F1
SQ
64.9/ 64.9/70.7
68.5/74.9 42.9/50.1
73.3/79.5 45,5/53.7
72.3/78.5 43.8/53.4
78.6/92.2 41.7/72.1
76.8/82.6 45.9/54.8
73.8/90.2 39.2/70.5
MuSiQue
CQ
34.9 9/44/
Table EM and scores of Answer and Support Passage on MuSiQue using different reader models. SQ represents simple question and CQ represents complex question.
reasoning type of the gold standard, thus failing to yield the correct final answer. To illustrate, the gold H-expression is a 3-Hop while our model produces a 2-Hop, it indicates a missed decompo- sition into two distinct questions. As result, the necessary context answer the question is absent. Addressing this issue in open-domain setting (Liu et al., 2021) could enhance the likelihood of accurately responding more intricate questions. As the number of hops increases, our model might face exposure bias (Bengio et al., 2015), where an error in one step can affect the final an- swer. However, once we identify the error's source, we correct it, whether it's incorrect H- expression or single-hop answer at H-execution stage. This exposure bias can also be mitigated using a beam search (Wiseman and Rush, 2016), generating multiple answers at each step and select- ing the highest-scoring one.
profound insights have been instrumental through-
out the course of this research.
Limitations
We acknowledge that our work could have the following limitations:
Even the defined H-expression can be used on various reasoning types and different text question answering datasets, it is not mature to be used to any type of reasoning. When the new reasoning type comes, we need to retrain the question parser. To solve the new reasoning type question, we plan take advantage of in-context learning in a large language model to generate H-expression as future work. It's worth mentioning that our executor can be casily adapted to new reasoning types by adding new symbolic rules and the reader network doesn' 't need to be retrained.
5 Conclusion
As mentioned in the error analysis section, the bottom-up question answering process could suffer from exposure bias since the next step question answering may depend on the previous predicted answers. To deal with this limitation, we anticipate that generating multiple answers using beam search in each step may greatly solve this issue. Since predicted candidates by current reader models have strong lexical overlap, general beam search needs to be revised provide a sufficient coverage of semantic meanings. We leave it for future work.
We propose HPE for answering complex questions over text, which combines the strengths of neural network approaches and symbolic approaches. We parse the question into H-cxpressions followed by the hybrid execution to get the final answer. Our extensive empirical results demonstrate that HPE has strong performance on various datasets un- der supervised, few-shot, and zero-shot settings. Moreover, our model has a strong interpretability exposing its underlying reasoning process, which facilitates understanding and possibly fixing its er- rors. By replacing our text reader with KB or Table based neural network, our framework can be ex- tended to solve KB and Table QA.
References
6 Acknowledge
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 39 49.
The authors extend their appreciation to the mem- bers of the Salesforce Al Research team for their constructive discussions. We also grateful to the anonymous reviewers for their invaluable feed- back. Special thanks are reserved for late Prof. Dragomir Radev, whose invaluable guidance and
Muhammad Shahzaib Baig, Azhar Imran, Aman Ullah Yasin, Abdul Haleem Butt, and Muhammad Imran Khan. 2022. Natural language to sql queries: A re- view. Technology, 4(1):147-162.

---

Samy Bengio, Oriol Vinyals. Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. Advances neural information processing systems, 28.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, 7ac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1777-1901
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781.
Christopher Clark, Kenton Lee, Ming- Ming-Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-2936.
Tushar Khot, Daniel Khashabi, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2021. Text modular networks: Learning to decompose tasks in the lan- guage of existing models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1264-1279.
Ruiliu Fu, Han Wang, Jun Zhou, and Xuejun Zhang. 2022. Na-reviewer: Reviewing the context to im- prove the error accumulation issue for multi-hop qa. Electronics Letters, 5866)2237-239
Takeshi Kojima. Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems. 35:22199- 22213.
Yu Gu, Vardaan Pahuja, Gong Cheng, and Yu Su. 2022. Knowledge base question answering: A semantic parsing perspective. arXiv preprint arXiv22209004994.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453- 466.
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2019. Neural module networks for reasoning over text. In International Conference on Learning Representations.
Veronica Latcinnik and Jonathan Berant. 2020. Explain- ing question answering models through text genera- tion. arXiv preprint arXiv:2004.05566)
Xanh Ho, KnhaKhoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Con- ference on Computational Linguistics, pages 6609- 6625.
Patrick Lewis. Yuxiang Wu, Linging Liu, Pasquale Min- ervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp. and Sebastian Riedel. 2021. Paq: 65 mil- lion probably-asked questions and what you can do with them. Transactions of the Association for Com- putational Linguistics, 9:1098-1115.
Gautier Izacard and Édouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 874-880.
Xi Victoria Lin, Richard Socher, and Caiming Xiong. 2020. Bridging textual and tabular data for cross- domain text-to-sql semantic parsing. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 4870-4888.
Harsh Jhamtani and Peter Clark. 2020. Learning to explain: Datasets and models for identifying valid reasoning chains in multihop question-answe answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 137-150.
Ye Liu, Kazuma Hashimoto, Yingbo Zhou, Semih Yavuz, Caiming Xiong, and S Yu Philip. 2021. Dense hierarchical retrieval for open-domain question swering. In Findings of the Association for Compu- tational Linguistics: EMNLP 2021, pages 188- 188-200.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- une 1: Long Papers), pages 1601-1611.
Ye Liu, Semih Yavuz, Rui Meng. Dragomir Radev, Caiming Xiong, and Yingbo Zhou. 2022. Uni parser: Unified semantic parser for question answering on knowledge base and database. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8868.

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Oner Levy, Mike Lewis. Luke Zet,lemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint
Yingyao Wang. Yongwei Zhou, Chaoqun Duan, Junwei Bao, and Tiejun Zhao. 2023, Hoppg: Self-iterative program generation for multi-hop question answer- ing over heterogeneous knowledge. arXiv preprint 13Xiv:3308.12577
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang. Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of- thought reasoning, arXiv preprint 1rXiv:2301.13377
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe V Le. Denny Zhou, et al. 2022. soning in large language models. In Advances in Neural Information Processing Systems.
prompting elicits rea-
Sewon Min, Victor Zhong, Luke Zettlemoyer, and Han- nanch Hajishirzi. 2019. Multi-hop reading compre- hension through question decomposition and rescor- ing. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 6097-6109.
Sam Wiseman and Alexander M Rush. 2016. Sequence- to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empiri- cal Methods in Natural Language Processing, pages 1296-1308
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's ' transformers: State-of- the-art natural language processing. arXiv preprint 191.v:1910.03771.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1 1-67.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gard- ner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understand- ing benchmark. Transactions of the Association for Computational Linguistics, 8: 188-198.
Pranav Rajpurkar. Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference Empiri- cal Methods in Natural Language Processing, pages 2369-238 -3800.
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-bass for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long Papers), pages 641-69 1651.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. Re- act: Synergizing reasoning and acting in language models. In NeurlPS 2022 Foundation Models for Decision Making Workshop.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022a. Interleaving retrieval with chain of ofotth reasoning for knowledge- intensive multi-step questions. arXiv preprint arXiv:2212.10509.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. Qa-gnn: Rea- soning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 335-546.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022b. Musique: Multi- hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554.
Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, Ni- tish Shirish Keskar, and Caiming Xiong. 2022. Mod- eling multi-hop question answering single se- quence prediction. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 974 990.
Oriol Vinyals, L ukasz Kaiser, Terry Koo, Slav Petrow, Ilya Sutskever, and Geoffrey Hinton. 2015. Gram- mar as a foreign language. In Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Bd H Chi, Sharan Narang, Aakanksha Chowdhery. and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2022. Rng-kbqu Generation augmented iterative ranking for knowledge base ques- tion answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 6032-6043.

---

Jiajie Zhang, Shulin Cao, Tingjia Zhang, Xin Lv, Jiaxin Shi, Qi Tian, Juanzi Li, and Lei Hou. 2023. Rea- soning over hierarchical question decomposition tree for explainable question answering. arXiv preprint arXiv:2305.15050
Wenting Zhao, Ye Liu, Yao Wan, Yibo Wang, Zhongfen Deng, and Philip S Yu. 2023. Localize, retrieve and fuse: generalized framework for free-form question answering over tables. arXiv preprint arXiv:2309.11045
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le, et al. 2022. Least-to-most prompting enables complex reasoning large language models. The Eleventh Interna- fional Conference on Learning Representations.

---

A H-expression Examples of Musique and 2WikiQA
first parses the complex question into H-expression. Then hybrid executor will convert the binary tree
Complex
party holds mejor
table and 10, we show the H-expression exam- plcs and parsing tree under each reasoning type of Musique and 2WikiQA.
Representatives, control of poltical body that President calls on support in his USAF appointments?
EndZEnd predicted answer. 1931
0
B Supervised Training Details
To train the question parser, initiate H-parser using T5-large model. We trained with batch size of 32 with learning rate of 3e-5 for 20 epochs on both MuSiQue and 2WikiQA. We se- lected the model weight based on evaluating the H-expression exact match. We base our reader net- work FiD on TS-large. We use 20 passages with maximum length of 256 tokens for input blocks MuSiQue dataset and 10 passages with 356 tokens as text length on the 2WikiQA dataset. We trained the reader model with batch size of 8 with a learning rate 5e-4 for 40k steps.
Lountoll the A1?
Senate
QL
41 the Republicand
02: Upon whom does the president call on for supportiin appointments the USAF?
QZ: Who hold the majority in representaaiies?
Figure 3hop2-type MuSiQue question example and how framework finds the final answer.
from the H-expression, where operation and natu- ral sub-question its nodes. H-executor parses the binary tree from the rightmost left node the left and upper layer with considering the operation.
Performance of each Different Reasoning Type
At each question node, the reader neural network will take sub sub-questions and multiple paragraphs input to generate the sub-answers. We store the sub-answer in memory for later substitution of the placeholder. For example, Q3 will be rewritten A1 with the answer QI (the Republicans) and A2 with the answer of Q2 (Senate) as the new question Q3' "when did Senate take control of the Republi- cans.. The final answer is obtained by answering Q3',
We represent the Answer Fl performance under different reasoning types both MuSiQue and 2WikiQA in Figure Our hybrid question parsing and execution model performs significantly better than directly getting the answer model in both QA showing that the advantage of delegating seman- tic parsing to solve complex textual questions. In MuSiQue, for the relevant simple reasoning types (2hop, 3hopl), our model outperform FiD by great margin. For complex reasoning types (3hop2, 4hop1, 4hop2 and 4hop3), our model gets lower performance compared with the simple reasoning types because the exposure bias issue becomes worse with the step of reasoning increase. But still has a equivalent or better perform comparing End-to-End FiD. In 2WikiQA, our model performs best on all four reasoning type. Especially on the frame-
0.75
score
0.55
Musique Reasoning
most complex type bridge comparison, work greatly outperform, which shows using deter- ministic symbolic representation is more robust to produce correct answer.
case study of how HPE reasoning
0.88
0.0
Irfenencc
ZWikiQA Roasoring Type
In Figure 4, we show an example that FiD pre- dicts a wrong answer but our model correctly pre- dicts. Given complex question, our framework
Figure 5: Answer score on each reasoning type on MuSiQue and 2WikiQA.

---

Who
county
National Park!
Hesgprosion: 2o0] Who the depety peine mimvae of Asoll, What country of Inagua National Pack
Whar is the prine misiatr Anall Whatis of Iaaqua National Park
H-ppp l-iitty
Questiou:: When the spock vellodto clanch sçli from the riligions inaiiution located the city whare the creatur of The Judganent died? city
Heeegrnstoon: great spllt
Infguerr
Thee:
Where the grekk entladto chanch spin Irom
In whur cit dil Ansti die
Who craate of The Last Jolgnent
Questio:: When the capiiol of Virginta mave fnom Mebert Banks Innilplax: the sown WIVR-F Mis lscosed in?
Heegrrssion: rots copttal viginii noved fron Uncoo|
licensed place
Thee:
Whene cld capiral of vaginn moved from Andfd Andil
Whaltown WTVR: FM lncoeedia! What place of bish Rlobert Banks
1- l-Iirity
Questiou: When the sivil war stas country whose sapial was lome to the mas alle wham Karelyo was namse?
Hesqpencton:
expisi cay cornyy. join
Kanolor What
Thee:
When cisil war Araki
Andit cegital city of which country 1818
Konijoo
Pype: d-Happ 3-Entiy
Question: Mustin orailes imadde cvenyy M-Mactumah
follwned MoMuntamid!
H-sgroosion zo0n| | When zid mudim amias unde Acoll and Anak2, Uxios What cxuntry of Al-Mastummh 2015] Wiss country citomnbipp of Arail, Al-Mis lomnid fellowed by Who:
When did masiim umioo ansak Anal3 und Ausi2
CODON
What country of citianolip Anshl
Al-Ma uamid fillowed What
Table 9: Examples of H-expression and parsing tree under each reasoning types in MuSiQue.

---

Type: Congurisiio
Question: Which film released first Who Kissing Me? Bush Christmas?
H-expression:
publicution And Who Me7, What publicution Bush Christmas]
COMP_E
What publication dase of Who Kissing
What publication date Bush Christmas?
Type: Bridge Compurison
Question: Which film has the director who was later, Slzepers East Leaving Pear Behind?
H-expression: coMP> JoiN When date of binhor #3, Who director af Sleepers Fist J. JOIN When is date of birth of #1, Who director of Lenving Fear Behind |
Tree:
COMP_>
JOUN
JOIN
When date of hirts of 43 Who director of Sleepers East
When is date of birth of f1l Who director of Leaving Fear Behind
Type:
Question: sibling-in-aa
H-expression: JOIN spouse
sibling Favila Asturias
JOIN
spouse sibling Asturias
Type: Congusiiiooal
Question: Where did the founder of University Of Flura die?
H Heeexpresiio:: JOIN Where is #1's place of death, The Universidad De Piura is founded by Who ]
Tree:
JOIN
Where is Al's place death
The Universidad De Piura is founded by Who
Table 10: Examples of H-expression and parsing under each reasoning types in 2WikiQA.


Reviews:
[{"review_id": "sbnHsTQpGD", "reviewer": "Reviewer_5EBV", "summary": "", "questions": "A: Line 508-510. Which LLM did you use as a baseline to compare in zero-shot performance? \nB:  When you get a wrong H-expression, how do you decompose it? Maybe I missed it, but is there any constraint for generating the H-expressions? Because if not, you may get a wrong H-expression which cannot be decomposed. \nC: For the model FiD LF->Ans, how do you extract the H-expressions? If it is the same way that you utilize for your model, then what is the difference between FiD LF->Ans and your model?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely thank you for the feedback and the issues you've raised. While we acknowledge and address the issues you've pointed out, we'd like to highlight that some of the major concerns have been covered in the paper and do not detract from our study's central contributions. We provide a detailed response to each point below (RR refers to \"Reason to Reject\", TGS refers to \"Typos Grammar Style\"):\n\n### RR 1: How do the model’s components contribute to the performance compared to other models?\nThank you for this question. In our paper, we present a synergistic integration of neural and symbolic approaches within a unified framework. We’d like to underscore the significance of distinct components across two key dimensions:\n\n1) Effectiveness of Combined Approach: A sole reliance on neural methods proves to be insufficient. In Table 2, our integrated neural-symbolic model, HPE, achieves a F1 score of 50.1 on the MuSiQue dataset. Conversely, when evaluating FiD(LF->Ans), a configuration where the H-expression serves as the input question and the network generates the answer, we observe underwhelming performance, yielding only a 44.8 F1 score on MuSiQue. This discrepancy underscores the tangible benefits brought about by the H-executor component, which operates as a potent symbolic processor.\n\n2) Robustness of H-parser: Detailed in Table 6 is an ablation study conducted on the H-parser module. When supplying gold H-expressions to the H-executor, the F1 scores reach 57.5 on MuSiQue and 86.5 on 2WikiQA. When we substitute the gold H-expressions with a T5-large H-parser, which generates H-expressions fed into the H-executor, the F1 scores decrease slightly to 50.1 on MuSiQue and 84.5 on 2WikiQA. This yields a difference of 7.4 on MuSiQue and 2.0 on 2WikiQA, indicating an acceptable margin between the outcomes derived from generated H-expressions and those gold labels.\n\n### RR2 & Question B: When you get a wrong H-expression, how do you decompose it?\n\nThat’s a good question. For H-parser, the executable rate of the greedy decoding H-expression is 95.6%. For further improvement, we use beam search in H-parser to generate a list of 10 H-expression candidates (mentioned in Line 240-242). Each H-expression is executed until we identify an executable one. In this way, the execution rate rises to 99.7% for the top 10, pointing to the robustness of our H-parser. To enhance clarity, we will include an executability analysis for Top-k H-expressions in the revised version.\n\n\n### Question A: Line 508-510. Which LLM did you use as a baseline to compare in zero-shot performance?\n\nTo illustrate our model's capability within a zero-shot context, we engaged in a comparative analysis of our methodology against the Standard and CoT prompting methodologies outlined in ReAct [1]. In these methodologies, the PaLM-540B model is utilized as the LLM. While our approach operates in a zero-shot paradigm, both the Standard and CoT strategies leverage few-shot in-context learning. Notably, our framework in a zero-shot setting, reliant solely on the 770M T5-large model, surpasses the few-shot results of Standard and COT with PaLM-540B when evaluated on the HotpotQA dataset. The success of our approach underscores the benefits of integrating neural networks with symbolic reasoning to answer complex questions. \n \n### Question C: the difference between FiD LF->Ans and your model\n\n“FiD LF->Ans” means that the FiD model takes H-expression along with multi-passage context as inputs, and subsequently generates the answer as a sequence of tokens (Seq2Seq). This mechanism contrasts with our method, which capitalizes on hybrid parsing (via the H-parser) and execution (through the H-executor).\n\n### TGS: need their conference citation instead of the arXiv version\n\nWe appreciate your suggestion on correcting the citation format. We will update all references and use the correct format in the revised manuscript.\n\nWe hope that our explanations effectively address your concerns and potentially lead to an adjustment of your initial assessment. Thank you for your consideration.\n\n[1]Yao, Shunyu, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. \"ReAct: Synergizing Reasoning and Acting in Language Models.\" In The Eleventh International Conference on Learning Representations. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Decomposition of complex questions into simpler explainable H-expressions. \n- Proposed method achieves better performance compared to SOTA and end-to-end methods.", "reasons_to_reject": "- Analysis into how the model's components contributed to the performance compared to other models \n- Limited explanation on how the approach deals with errors from the semantic parsing model", "questions_to_authors": "A: Line 508-510. Which LLM did you use as a baseline to compare in zero-shot performance? \nB:  When you get a wrong H-expression, how do you decompose it? Maybe I missed it, but is there any constraint for generating the H-expressions? Because if not, you may get a wrong H-expression which cannot be decomposed. \nC: For the model FiD LF->Ans, how do you extract the H-expressions? If it is the same way that you utilize for your model, then what is the difference between FiD LF->Ans and your model?", "paper_topic_and_main_contributions": "This paper addresses the problem of complex question answering by applying an intermediate step which decomposes the complex question into simple logical expressions. Their approach consists of a H-parser which: 1) generates an H-expression and 2) decomposes the H-expression into simpler statements that can be executed in order. Afterward, they utilize a reader network to extract the answer for each subquestion. \n\nThe approach shows better results than SOTA our end to end problems, and the authors also analyze zero shot performance. However, the significance of the results it is not clear, and which component brings the improvement. The components of the model are pretrained in other datasets, and it is not clear if the other models are also pretrained in the same datasets. For example, FiD + Pretraining, has better performance than FiD + H-expressions.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Several references need their conference citation instead of the arXiv version. For example:\n- Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. arXiv preprint arXiv:1803.06643. **===>** Alon Talmor and Jonathan Berant. 2018. The Web as a Knowledge-Base for Answering Complex Questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651, New Orleans, Louisiana. Association for Computational Linguistics.\n\n2. Related work section needs to be expanded"}}, {"review_id": "Kcj0cWhneH", "reviewer": "Reviewer_CHKJ", "summary": "", "questions": "How can the framework adapt to new reasoning types or datasets without retraining the parser? Could in-context learning help here?\n\nHave the authors experimented with generating multiple candidate answers per step using beam search? How well does it address exposure bias?\n\nThe paper mentions replacing the text reader with a KB or table reader to extend this framework. Have the authors experimented with this? If so, how well does the framework adapt to KBQA or text-to-SQL semantic parsing tasks?\n\nHow does the framework compare to recent work on prompt-based reasoning for complex QA? Could prompting help improve the parser?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for your detailed review and feedback. \n### Question 1: how can the framework adapt to new reasoning types or datasets without retraining the parser? Could in-context learning help here?\n\nThank you for the insightful question. In our HPE framework, we use a T5-base or T5 large as the H-parser, which has limited model capability compared to LLMs. In this setting, to accommodate new reasoning types, we acknowledge that our current model demands retraining. Indeed, our neural-symbolic approach can be used with LLMs, where we believe adding new reasoning types could be much easier with no retraining or solely prompt tuning. Although covering the full aspects of complex reasoning through LLMs is beyond this paper's scope, we’d like to share some of our proposals here. \n\n1) In-context learning. Yes, we believe in-context learning could help with H-expression generation. We could employ a prompt starting with the parsing task description, followed by the H-expression grammar and several demonstration examples. This setup enables the LLM to generate new H-expressions for complex questions without needing to retrain the model. We did some experiments exploring the potential of generating H-expression with in-context learning. Our result suggests that although LLM can proficiently generate basic H-expressions like 2-hop structures, it encounters limitations when dealing with more advanced composition reasoning types like 4-hop. The following is an example of in-context learning in LLMs.\n\n2) Prompt tuning. To obtain a more specified LLM for question answering in a neural-symbolic way, we could add every operation of the corresponding reasoning type as additional soft prompt tokens.  In this way, we could learn the operations as the prompts, rather than retraining the whole model. \n\nInput template to ChatGPT for linear-chain and intersection reasoning:\n```\nConvert question -> H-expresion: question is the multi-hop complex question. And the H-expression contains operators and elements. Each element is a natural question. Operator is from [JOIN, AND] and each of them is a binary operator, which means it contains two elements. JOIN(QA(a) ; QA(b)) operator is for linear-chain reasoning that question a contains the placeholder (e.g. Ans#1) of question b's answer. Placeholder is to represent the answer from previous question. AND(QA(a) ; QA(b)) operator is for intersection reasoning that question a and b can be processed parallel. \nJOIN and AND can be compositioned, like JOIN(QA(c), JOIN(QA(b), QA(a))), AND(QA(c), JOIN(QA(b), QA(a))). Which composition to use, please follow the demo examples. \nFor example:\nquestion: Who plays the creator of Sunflowers in Doctor Who?\nH-expression: JOIN( QA( who plays Ans#1 in doctor who ) , QA( Who is creator of Sunflowers ) )\nanother 5 demos\nGiven question: XXX, \nH-expression:\n```\nHere are two real outputs by ChatGPT. \n\n***Good example:***\n\n> ***Given question:*** At the end of what year did the tribes from the place where the Rossen culture is located invade the Roman Empire?\n\n> ***Gold H-expression:*** JOIN( QA( At the end of what year did the tribes from Ans#1 invade the Roman Empire? ), QA( Where is the Rossen culture located? ) )\n\n> ***ChatGPT Generated H-expression:*** JOIN( QA( At the end of which year did Ans#1 tribes invade the Roman Empire? ) , QA( Where is Rössen culture located on terrain feature? ) )\n\n\n***Wrong example:***\n\n>***Given question:*** What is the longest home run in the history of the league that Jim Wilson's team is a part of?\n\n>***Gold H-expression:*** JOIN( QA( what is longest home runs in Ans#2 history? ) , JOIN( QA( What league was Cleveland Ans#1?) , QA( Jim Wilson is the member of which sports team?) ) )\n\n>***ChatGPT Generated H-expression:*** JOIN(QA(What is the longest home run in the history of the league that Ans#2's team is a part of), QA(Who is Jim Wilson))\n\n### Question 2: Have the authors experimented with generating multiple candidate answers per step using beam search? How well does it address exposure bias?\n\nYes, we extended our exploration beyond beam search solely in the H-parser, delving into its integration within the H-executor as well. This approach involves utilizing beam search to generate a selection of candidate answers from the single-hop reader. In the context of HotpotQA, for instance, we applied beam search to generate the Top-10 answers, subsequently implementing a majority vote mechanism for each step. The results of this strategy led to an elevated performance from 32.6 to 33.9. This initial observation indicates that refining answer quality at each discrete step can effectively mitigate exposure bias. \n\nWhile the end-to-end model may not inherently suffer from exposure bias, its performance often degenerates when dealing with complex tasks due to the obscured nature of its reasoning process. Correcting an erroneous response to a complex question within an end-to-end model can be challenging. Conversely, our hybrid framework, intertwining neural and symbolic components, offers an explicit expression of the question through H-expression. Consequently, in instances of incorrect final answers, we are adept at pinpointing the erroneous sub-step of reasoning, facilitating precise corrections. The experiments and the discussions about exposure bias will be added in the revision.\n\n\n### Question 3: The paper mentions replacing the text reader with a KB or table reader to extend this framework. Have the authors experimented with this? If so, how well does the framework adapt to KBQA or text-to-SQL semantic parsing tasks?\n\nWe haven't directly tailored our framework for KBQA or text-to-SQL tasks. Instead, we're delving into a multi-source complex question-answering approach. In this paradigm, one hop might draw from textual sources for answers, whereas the next could tap into a KB or table. One of our primary challenges is the absence of a comprehensive benchmark to assess the model's performance in such settings. Our ongoing efforts are concentrated on establishing this benchmark and investigating the potential of integrated QA engines.\n\n### Question 4: How does the framework compare to recent work on prompt-based reasoning for complex QA? Could prompting help improve the parser?\n\nIn comparison to recent prompt-based reasoning approaches, such as CoT, our HPE framework shares a common methodology in decomposing complex questions into single-hop questions. However, a distinguishing feature of our approach is the utilization of symbolic operations during the reasoning process, ensuring that HPE offers more deterministic outcomes. This characteristic reduces the risk of LLM hallucinations, making our answers more reliable. When evaluated in a zero-shot setting on HotpotQA against CoT, our model, built on the 770M T5, outperforms with a score of 32.6, compared to CoT's 29.4, which uses the larger 540B PaLM. Additionally, as mentioned in Answer 1, in-context learning and prompt tuning could facilitate the parser to add new reasoning types without fully retraining.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Pros:\n\nAchieves new SOTA results on complex QA datasets MuSiQue and 2WikiMultiHopQA, outperforming prior neural and prompting-based methods.\n\nShows stronger generalization ability in few-shot and zero-shot settings compared to end-to-end neural models like FiD. This modularity helps disentangle parsing complex questions from answering simple ones.\n\nQuantitative results show the framework generalizes better with less training data. This could make it more adaptable with limited labeled data.\n\nProvides interpretability by exposing the explicit reasoning process through H-expressions. This facilitates understanding and fixing errors.", "reasons_to_reject": "Cons:\n\nThe set of symbolic reasoning operations may not cover the full range of complex reasoning required for open-domain QA. Expanding the operations would require retraining the parser.\n\nThe parser relies on training data to generate valid H-expressions. It may fail on new reasoning types without retraining.\n\nExposure bias during multi-hop reasoning - an error in one step affects subsequent steps. Beam search could help mitigate this issue.\n", "questions_to_authors": "How can the framework adapt to new reasoning types or datasets without retraining the parser? Could in-context learning help here?\n\nHave the authors experimented with generating multiple candidate answers per step using beam search? How well does it address exposure bias?\n\nThe paper mentions replacing the text reader with a KB or table reader to extend this framework. Have the authors experimented with this? If so, how well does the framework adapt to KBQA or text-to-SQL semantic parsing tasks?\n\nHow does the framework compare to recent work on prompt-based reasoning for complex QA? Could prompting help improve the parser?", "paper_topic_and_main_contributions": "\nThis paper proposes a hybrid framework (HPE) that combines neural networks and symbolic reasoning for answering complex questions over text. The key ideas are:\n(1) Parse questions into an intermediate representation called H-expression, which hierarchically combines simple questions using symbolic operations like JOIN, UNION, etc.\n(2) Execute the H-expression using a hybrid executor - use a neural reader to answer simple questions and symbolic rules to deterministically aggregate the answers.\n\n\n\n\nLimitations:\n\nLimited coverage of reasoning types. New types require retraining the parser.\nExposure bias in multi-hop reasoning. Beam search could help but needs more exploration.\nOverall, this is a novel and promising direction bringing together the strengths of neural and symbolic reasoning. The results demonstrate strong performance and the modular design provides interpretability. Addressing the limitations around adaptability and exposure bias could make this approach more robust and widely applicable.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KDp9MhfrpT", "reviewer": "Reviewer_NMo5", "summary": "", "questions": "1.  Does the non-PT version of HPE still use TriviaQA, SQuAD and BoolQ? What is the performance of an SA (PT) if you can train such a model?\n2. Could you provide additional details about the CoT result on zero-shot HotPotQA? Were there exemplars used, use of self-consistency, instruction-tuned model, etc.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for your detailed review and feedback. We believe there might have been some misunderstandings which we hope to clarify in this rebuttal.\n\n### Question 1(a): Does the non-PT version of HPE still use TriviaQA, SQuAD and BoolQ?\n\nSorry that our symbols cause your confusion. We clarify that **these datasets were not utilized in the non-PT version of HPE**. Our multi-passage single-hop reader PT process has two steps. First, it uses single-passage single-hop PAQ to train a T5 model. This tuned T5 model then serves as an initialization for FiD, which undergoes further training on the multi-passage single-hop datasets TriviaQA, SQuAD, and BoolQ. Therefore, when we refer to the \"non-PT version\", it implies a version untouched by either PAQ or the other three datasets. More pre-training details on the reader network can be found in Section 4.1.3 and Appendix B.\n\n### Question 1(b): What is the performance of an SA (PT) if you can train such a model?\n\nThank you for pointing out this question. We integrated our pre-training strategy using the same collection of PT single-hop QA datasets for SA. More precisely:\n* We first pre-trained the SA model on PAQ and TriviaQA, SQuAD and BoolQ.\n* We fine-tuned the SA model on MuSiQue.\n\nHowever, intriguingly, the resulting model’s performance is 51.8, which is worse than non-PT 52.0(reproduced by us). This suggests that pre-training on single-hop QA datasets may impair the SA model's performance on MuSiQue. One plausible explanation is that single-hop QA tasks do not strengthen the ability of complex multi-hop reasoning required by MuSiQue. This observation amplifies the significance of our approach of offloading the complex reasoning to hybrid parsing and execution, while single-hop QA engine is all it needs. We will include these findings in our finalized version.\n\n###  Question 2: Could you provide additional details about the CoT result on zero-shot HotPotQA? \n\nWe report the CoT results from ReAct[1], which uses PaLM-540B as the LLM. Both Standard and CoT use few-shot in-context learning. For CoT with self-consistency, ReAct reported a score of 33.4. Notably, self-consistency serves as a general decoding upsampling approach that can also be applied to our HPE framework by generating multiple answers in each single-hop reader. Notably, HPE built on 770M T5-large obtains a score of 33.9 when coupled with self-consistency, which outperforms self-consistency CoT with PaLM-540B. We will incorporate these details in the finalized version. \n\nIn conclusion, we are grateful for the depth and precision of your feedback. Such insights guide us in enhancing the clarity and completeness of our work. We hope that the clarifications provided address your concerns.\n\n[1]Yao, Shunyu, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. \"ReAct: Synergizing Reasoning and Acting in Language Models.\" In The Eleventh International Conference on Learning Representations. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In my opinion, the primary strength of the work is its ability to provide an interpretable chain of reasoning in the zero-shot setting at a competitive level of performance with End2End approaches and potentially better than any other approach which can provide a chain of reasoning.\n\nThe paper is written well.", "reasons_to_reject": "I believe there are two main problems with this work -\n\nFirst, when measuring the performance against SOTA in MuSIQue and 2WikiQA, which is SA (Trivedi et al 2022b), there is a possible source of difference which does not relate to the proposed method: the additional datasets used for training the reader, which include PAQ, TriviaQA, SQuAD and BoolQ. I believe that even the non-PT version of HPE uses the latter 3 datasets, please refer to question (1). Thus, a fair comparison of performance would be SA (PT) vs HPE (PT); and looking at SA vs HPE, I suspect the performances would be more on-par. The fact that FID does just as well on zero-shot HotPotQA as HPE seems to indicate that the H-expressions might not be providing any significant performance improvement.\n\nIf that is the case, then the main contribution of this work is not the performance improvement but the interpretable chain of reasoning this method provides. If so, its main 'competitor' would be the Chain-of-Thought (CoT) approaches and its more recent advancements, which are not done justice in this work (See question 2). While theoretically HPE's chain of reasoning is deterministically executed while no such guarantee is present in LLM's, it is unclear how much of a practical problem this is. I strongly advocate for an empirical comparison against a best-effort CoT baseline to strengthen your work - meaning CoT with Self-Consistency, in-context exemplars from MuSIQue and 2WikiQA, and ideally with an instruction-tuned LLM.", "questions_to_authors": "1.  Does the non-PT version of HPE still use TriviaQA, SQuAD and BoolQ? What is the performance of an SA (PT) if you can train such a model?\n2. Could you provide additional details about the CoT result on zero-shot HotPotQA? Were there exemplars used, use of self-consistency, instruction-tuned model, etc.", "paper_topic_and_main_contributions": "This work proposes a new approach called the Hybrid Parsing and Execution (HPE) framework for answering complex questions using text. It consists of a network to decompose the complex question into a semantic parse called the H-expression; where each primitive in the expression is a simple (single-hop) query. A second reader network then answers each simple question and the H-expression is resolved recursively bottom-up.\n\nIt seems to achieve SOTA on the MuSIQue and 2WikiQA datasets, as well as demonstrate strong performance in the few-shot and zero shot settings in HotPotQA.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3QGsh1XFIH", "reviewer": "Reviewer_YSXk", "summary": "", "questions": "A: The evaluation of the H-parser is based on the exact match score and F1, are there any other metrics worth to be considered?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful feedback. Please find our detailed responses to your comments below (RR refers to “Reason to Reject”):\n\n### ***RR 1: If the H-parser makes a mistake in generating the H-expression, it could lead to an incorrect final answer even if the H-executor works perfectly.***\n\nWe understand the concern regarding potential error propagation inherent to pipeline approaches. To provide some clarity, our H-parser exhibits a robust executable rate. Specifically, the top-1 execution rate stands at 95.6%, and with beam search, this rate rises to 99.7% for the top 10 H-expressions. It's pertinent to note that the primary discrepancy from the gold H-expression often involves a differing reasoning path. For example, a 3-hop reasoning path in the gold might be represented as a 2-hop in our generated H-expression. It leads to a problem that among all the candidate passages, there is a chance that none of them can be helpful in answering the question. As part of our ongoing research, we're looking into enhanced retrieval models and refined question decomposition techniques to mitigate this issue.\n\n\n### ***RR 2: The H-expression currently consider eight types of operations. While these cover a wide range of question types, there could be complex questions that cannot be represented using these operations.***\n\nWe recognize the limitation that the eight operations may not represent all possible complex questions. While our approach synergizes both neural and symbolic methods to address the question-answer task, capturing every facet of complex reasoning extends beyond this study's objectives. We’d like to share some of our proposals here. \n\n1) In-context learning. We think LLMs, when provided with suitable prompts, can help expand our model's capability to generate new types of H-expressions. Our initial experiments suggest that LLMs can efficiently generate basic 2-hop H-expressions, but face challenges with more complicated structures like 4-hop expressions.\n2) Prompt Tuning. To obtain a more specified LLM for question answering in a neural-symbolic way, we could add every operation of the corresponding reasoning type as additional soft prompt tokens.  In this way, we could learn the operations as the prompts and easily adapt to the new complex reasoning tasks.\n\nBoth of these strategies offer a path toward making our framework more adaptable to various complex reasoning tasks.\n\nInput template to ChatGPT for linear-chain and intersection reasoning:\n```\nConvert question -> H-expresion: question is the multi-hop complex question. And the H-expression contains operators and elements. Each element is a natural question. Operator is from [JOIN, AND] and each of them is a binary operator, which means it contains two elements. JOIN(QA(a) ; QA(b)) operator is for linear-chain reasoning that question a contains the placeholder (e.g. Ans#1) of question b's answer. Placeholder is to represent the answer from previous question. AND(QA(a) ; QA(b)) operator is for intersection reasoning that question a and b can be processed parallel. \nJOIN and AND can be compositioned, like JOIN(QA(c), JOIN(QA(b), QA(a))), AND(QA(c), JOIN(QA(b), QA(a))). Which composition to use, please follow the demo examples. \nFor example:\nquestion: Who plays the creator of Sunflowers in Doctor Who?\nH-expression: JOIN( QA( who plays Ans#1 in doctor who ) , QA( Who is creator of Sunflowers ) )\nanother 5 demos\nGiven question: XXX, \nH-expression:\n```\nHere are two real outputs by ChatGPT. \n\n***Good example:***\n\n> ***Given question:*** At the end of what year did the tribes from the place where the Rossen culture is located invade the Roman Empire?\n\n> ***Gold H-expression:*** JOIN( QA( At the end of what year did the tribes from Ans#1 invade the Roman Empire? ), QA( Where is the Rossen culture located? ) )\n\n> ***ChatGPT Generated H-expression:*** JOIN( QA( At the end of which year did Ans#1 tribes invade the Roman Empire? ) , QA( Where is Rössen culture located on terrain feature? ) )\n\n\n***Wrong example:***\n\n>***Given question:*** What is the longest home run in the history of the league that Jim Wilson's team is a part of?\n\n>***Gold H-expression:*** JOIN( QA( what is longest home runs in Ans#2 history? ) , JOIN( QA( What league was Cleveland Ans#1?) , QA( Jim Wilson is the member of which sports team?) ) )\n\n>***ChatGPT Generated H-expression:*** JOIN(QA(What is the longest home run in the history of the league that Ans#2's team is a part of), QA(Who is Jim Wilson))\n\n### ***Question: The evaluation of the H-parser is based on the exact match score and F1, are there any other metrics worth to be considered?***\nBeyond the exact match score and F1 metrics, we've also considered BLEU and ROUGE for evaluating the H-parser's output. Specifically, we observed BLEU-2 at 79.20, BLEU-4 at 63.09, ROUGE-2 at 70.64, and ROUGE-L at 83.66 with the T5-large model. While these scores provide valuable insight into the H-parser's performance, we chose to emphasize the Answer EM and F1 metrics in our report, as they offer a more rigorous representation of our system's efficacy. We will add the BLEU and ROUGE results in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors propose a new framework that combines the strengths of neural networks and symbolic methods. This design allows for a more structured and interpretable approach to question answering.\n- This paper presents extensive experiments demonstrating that the HPE framework outperforms existing approaches in various settings, including supervised, few-shot, and zero-shot scenarios.\n- The authors show that their framework can be applied to multiple datasets, suggesting that it has broad applicability and can handle a wide range of complex questions.", "reasons_to_reject": "- My main concern is that the pipeline approach, while offering interpretability, can lead to error propagation. If the H-parser makes a mistake in generating the H-expression, it could lead to an incorrect final answer even if the H-executor works perfectly. The paper does propose some solutions to this issue, but it remains a fundamental challenge of the pipeline approach.\n- The H-expression currently consider eight types of operations. While these cover a wide range of question types, there could be complex questions that cannot be represented using these operations.\n", "questions_to_authors": "A: The evaluation of the H-parser is based on the exact match score and F1, are there any other metrics worth to be considered?", "paper_topic_and_main_contributions": "The paper introduces a new framework, Hybrid Question Parser and Execution (HPE), designed to answer complex questions over text. This approach merges the capabilities of neural networks and symbolic methods, enhancing both interpretability and generalizability in textual question answering. HPE framework is composed of two primary components: the H-parser and the H-executor. The H-parser transforms complex questions into an intermediate representation known as an H-expression, which includes primitives (single-hop questions) and operations (symbolic relationships between primitives). The H-executor then processes the H-expression using a blend of deterministic rules and a neural reader network to answer the decomposed simple questions. The authors conduct comprehensive experiments on multiple datasets, demonstrating that the HPE framework surpasses existing methods in supervised, few-shot, and zero-shot settings. The framework also improves interpretability and reveals the underlying reasoning process.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Problem: This paper addresses the problem of complex question answering.

Solution: It applies an intermediate step that decomposes the complex question into simple logical expressions. Their approach consists of an H-parser which: 
 1) generates an H-expression, hierarchically combining simple questions using symbolic operations like JOIN, UNION, etc.  
 2) Afterward, they utilize a reader network to extract the answer for each subquestion.


Here are the notable points of concern: 

- More extensive analysis of the factors contributing to the model's success. The authors elaborate on this, and I suggest they incorporate all these details in their revised paper. 
- How does the approach deal with errors from the parser? While the authors provide some statistics, the answer to this question is unclear. 
- Given the model's reliance on supervised data, expanding the model to tackle more diverse questions would require retraining the parser.
- My addition to this list: The paper has no 2023 citations, which is odd. Please revise their draft with all the relevant work published this year. 


All reviewers have assigned high soundness scores and overall buy the outcome. However, given the limited excitement about this work, I recommend the "findings" track.
Decision: Accept-Findings