Gradually Excavating External Knowledge for Implicit Complex Question Answering
Authors: Chang Liu
Xiaoguang Li
Lifeng Shang
Xin Jiang
Qun Liu
Edmund Y. Lam
Ngai Wong
Venue: EMNLP 2023 Findings (2023)
Keywords: Question answering
Knowledge Retrieval
Multi-step question answering
Large Language Model

Abstract: Recently, large language models (LLMs) have gained much attention for the emergence of human-comparable capabilities and huge potential. However, for open-domain implicit question-answering problems, LLMs may not be the ultimate solution due to the reasons of: 1) uncovered or out-of-date domain knowledge, 2) one-shot generation and hence restricted comprehensiveness. To this end, this work proposes a gradual knowledge excavation framework for open-domain complex question answering, where LLMs iteratively and actively acquire extrinsic information, then reason based on acquired historical knowledge. Specifically, during each step of the solving process, the model selects an action to execute, such as querying external knowledge or performing a single logical reasoning step, to gradually progress toward a final answer. Our method can effectively leverage plug-and-play external knowledge and dynamically adjust the strategy for solving complex questions. Evaluated on the StrategyQA dataset, our method achieves 78.17% accuracy with less than 6% parameters of its competitors, setting new SOTA in the ~10B LLM class.

Gradually Excavating External Knowledge for Implicit Complex Question Answering
Chang Liu', Xiaoguang Li", Lifeng Shang', Xin Jiang', Qun Liu',
Edmund Y. Lam', ,gai Wong'
'The University of Hong Kong, Huaawei Noah's Ark Lab 1con7êconnect. hku. (1 (lixiaoguang11, Shang.Lifeng Jiang. Xin, qun. gin.liujehuawei.com {elam, nwong)@eee. hku. hk
Abstract
also be multi-step and implicit, consisting of multi- ple sub-questions that cannot be directly identified from the question language. but require logical reasoning to form a problem-son solving strategy. Be- cause of the above challenging characteristics, how to answer open-domain implicit complex questions remains open question.
Recently, large language models (LLMs) have gained much attention for the emergence of human-comparable capabilities and huge po- tential. However, for open-domain implicit question-answering problems, LLMs may be the ultimate solution due to the reasons of: 1) uncovered out-of-date domain knowledge, 2) one-shot generation and hence restricted com- prehensiveness. To this end, this work pro- poses gradual knowledge excavation frame- work for open-domain complex question an- swering, where LLMs iteratively and actively acquire external information, and then reason based on acquired historical knowledge. Specif- ically, during each step the solving process, the model selects action to execute, such as querying external knowledge or performing a single logical reasoning step, to gradually progress toward final answer. Our method can effectively leverage plug-and-play external knowledge and dynamically adjust the strategy for solving complex questions. Evaluated on the StrategyQA dataset, our method achieves 78.17% accuracy with less than 6% parame- ters of its competitors, setting new SOTA for ~10B-scall LLMs.
For example, in the upper part of Fig. 1, an im- plicit complex question "Did any citizen of San Antonio vote for Boris Johnson" confuses the LLM because there is direct information about in- dividual voting history. However, the strategy of searching voting history is invalid does mean the question unsolvable. On the top right corner of Fig. 1, the question can decomposed into sub- questions about Boris Johnson and San Antonio, respectively, and a strategy of checking citizenship can be casily unsealed from the background knowl- edge (marked in red). Following the strategy, the answer is straightforward because US citizens not vote in UK elections. Now if the question is re-asked with the hint of "citizenship contradiction's the LLM can successfully recall its inner knowl- edge about Boris Johnson and San Antonio, hence correctly answering the question. However, differ- ent from existing works that try to design manual prompts (Wei et al., 2022b; Lyu et al., 2023) to serve the hint in the above example, want to stress that this approach is not always valid. Be- cause the key "citizenship contradiction" is not di- rectly linked the question text, but relies on the gradually increased knowledge during the solving process (e.g., background about Boris Johnson and San Antonio).
1 Introduction
Recently, powerful LLMs such as ChatGPT, GPT4 (OpenAI, 2023), PaLM (Anil et al., 2023), LLaMA and its variances (Touvron al.. 2023;
Taori et al., 2023), exhibiting human-alike ability in conversation. It is believed the LLMs memorize knowledge in their parameters from the vast pre- training data (Moiseev et al., 2022; Roberts et al., 2020). Nonetheless, they could still fail to solve open-domain implicit complex questions.
Another problem for solving plex questions with LLMs is the finite pretrained knowledge. In the bottom part of Fig. 1, we rewrite the question with two less well-known entities out of the LLM's knowledge scope, and it fails again due to limited knowledge.
com-
In real-world applications, users might ask ques- tions in arbitrary domain that requires specific knowledge, and expect the model return not only syntactically fluent but also factually correct an- swers. Beyond open-domain, the questions can
Therefore, in this work, we propose a pipeline

---

"Did any citizen of San Antonio vote for Boris Johnson?
[1] Who Boris Johnson?
(2) Where is San Antonio?
don't 't have access to personal data or real-time
information about indiviuaal vating cholces...
X
Boris Johnson is the
San Antonio city
Prime Minister of the UK.
Texas in the United States of America.
San Antonio: City, Texas USA
"iin: Cittennhiip controdiction"
Johnson: Prime Minister
(Strategy: Citizenship Contradiction)
(3) Are American citizens allowed to vote in the UK elections?
Son Antonio a city located in Texms, United States, Bons Johnson who serves as the Prime Minister of the United Kingdom.
politicuan
Only UK commonwealth citizens may elections.
highly uniikely that citizens of San Antonio would have directly voted Baris Johnson, as they eligible to vote UK elections.
this information,
The onswer No.
"Did any citizen of Bahadurabad vote for Alsin-Gioro Yizhu?"
Bohadurobad region Pokiston
???
Alsin-Gioro Yizhu oncient Chinese emperor Emperor does not come from election
apologize, couldn't find any information about
specific condidate named Alsin-Gioro Yizhu in relation elections political events.
(Strotegy: Political system)
Figure I: LLMs fail to solve open-domain complex questions due to unrecognized entities and implicit strategies. (1) the upper part, the LLM fails to answer the question with the one-shot generation, for there no off the-shelf answer or evidence to this question. However, the question can be decomposed into several sub-questions and be solved once the citizenship contradiction is identified. If the hint of "citizenship contradiction' is also given, the LLM successfully solve the question with the inner knowledge now. (2) But for the bottom case with less well-known entities, the LLM fails again due to lack of specialized knowledge about Misin-Gion Yizhu' and hence rejects to answer. Moreover, the strategy of the 'political system' not likely to be discovered from the question text only, unless enough knowledge is provided. 'Citizenship contradiction' is also possible solution.
named Gradually Excavating External Knowledge (GEEK) to address the main challenges of open- domain implicit complex question answering: ex- ternal knowledge, multi-step complexity, and im- plicit logic strategy. Given an open-domain multi- step implicit question, GEEK progressively decom- poses the problem into several sub-questions, and iteratively calls different modules for answering the sub-questions. In the end, a final answer con- cluded, synthesizing the historical sub-questions and their corresponding answers. Specifically, GEEK consists of three modules, core model, re- triever, and extractor. The core model handles logical reasoning and selects action to perform each time step, planning the solving strategy purposefully. The retrievers allocate relevant con- text paragraphs from the external corpus (e-g.. Wikipedia) to provide trustworthy knowledge, and the extractor condenses the textual knowledge into brief fact sentences.
GEEK to branch out different sub-questions during the solving process, thus exploring a strategy space and improving the final accuracy.
We verified GEEK on the challenging Strate- gyQA dataset (Geva et al., 2021), which consists of open-domain multi-step step implicit questions. GEEK achieves 77% accuracy with less than 6% param- eters of its competitors, refreshing the new SOTA for LLMs under 330B scale.
Our main contributions threefold:
We propose GEEK, a novel pipeline to solve open-domain complex questions by progres- sively acquiring external knowledge and ad- justing its strategy.
GEEK is able explore strategy space solve the question with different approaches, hence improving the overall performance.
During intermediate steps, GEEK can adjust the rest sub-questions based on the gradually increased external knowledge, hence forming valid strat- egy like in Fig. Considering there usually exist multiple valid solutions for a question, we enable
Our method is evaluated on the challenging StrategyQA benchmark, achieving 77.73% ac- curacy, surpassing vanilla LLMs such as Chat- GPT with 94% parameters less.

---

2 Related Work
ing iteratively reasoning for multi-step question answering only focus on datasets such as Hot- potQA (Yang et al., 2018), which do not involve implicit questions. Instead, their method assumes a straightforward direction from one step to another. For example, IRCoT (Trivedi et al., 2022) directly uses the 'thought' (intermediate result) from the last step the query for next-step retrieval, which highly relies on the strong connections between the entities from each step. Different from the above- listed methods, our GEEK modeling the process of purposely excavating external knowledge by com- posing sub-questions, and formulating a complete strategy gradually during the solving process.
Retrieving external information is widely adopted method that can provide flexible knowl- edge to extend LLMs for specific-domain tasks. (Izacard and Grave, 2020a) leverages retriev- ers (c.g., BM25 (Robertson et al., 1995) or DPR (Karpukhin et al., 2020)) to collect relevant passages, and fuse them in the decoder for a fi- nal answer. Similarly, (Zhu et al., 2021) proposes AISO, which performs multi-round retrieval with different retriever models of BM25, DPR, and LINK. It then synthesizes the retrievals for com- prehensive conclusion. HopRetriever (Li et al., 2021) adopts a multi-hop manner, which identifies significant entities in previous retrievals, and then uses the entities as queries for next step retrieval.
3 Problem definition
In this work, we focus on open-domain implicit complex question answering. Given a question q. the model is asked to derive the final answer z. Specifically, q is open-domain and hence requires some certain background facts to solve, denoted as F {fi). The facts come from an external corpus (e.g.. Wikipedia), not necessarily included in the model's pretraining dataset. The question & is also multi-step, meaning that it can be decomposed into several decomposition questions D {. Each d corresponds to a background fact ft. Noted that D is usually implicit from g, which means some d, can only be formulated until enough facts F, C F uncovered. Under the GEEK scenario, we define the question state Q at step t as:
Multi-step Implicit Question Answering involves complex questions that consist of several single- step questions, whose answers can be directly founded in reference context or inferred via logical deduction. The question is implicit if the decompo- sition strategy cannot be formulated merely from the question text. StrategyQA QA (Geva et al., 2021) is dataset for multi-step implicit question answering. including
solutions in the form of decomposition questions and corresponding fact sentences from Wikipedia. However, while hu- man beings can achieve 87% accuracy (Geva et al., 2021), the dataset has proved to be very challeng- ing for language models to solve. Merely improv- ing the quality of retrieval (Liang et al., 2022) or decomposition strategy (Katz et al., 2022) cannot effectively boost final answer accuracy.
Q: - (q. f1). fl... (d--1, (de, fo))
Several previous studics adopt the design of it- erative retrieval and reasoning to solve multi-step complex questions. IRGR (Ribeiro et al., 2022) performs iteratively retrieval to search for suitable premises. ReAct (Yao et al.) lets the model first generate a reasoning sentence about what next ac- tion (e.g., search via web API) to be performed, and
where the current decomposition d, and fact fo may may not have been decided yet. Q includes the question g and historical exploration steps.
Gradually Excavating External Knowledge
then execute the selected action. Maieutic Prompt- ing (Jung et al., 2022) introduces a maieutic tree that recursively entails component statements, and uses the concept of 'logical integrity' to verify each step. RR (He et al., 2022) combines CoT (Wei et al., 2022b) with retrieval to verify the correctness of the reasoning process, and achieved an accuracy of 77.73% on the StrategyQA dataset, the SOTA for LLMs below ~300B scale at that time. Nonetheless, most of the previous studies us-
In this section, we introduce GEEK for open- domain complex question answering. GEEK con- sists of three components, the core model, retriever and extractor model. Iteratively, the core model selects actions to perform from an action space A, conditioned on the current question state Q. Then the selected action executed and the question state is updated, gradually accumulating external knowledge until a final answer can be drawn.

---

Select Action
Qc
Legend
Neural inference
Q+
Select Action
Final Answer
Symbolic inference
Action Instruction
CoreModel
A
Imput output
execctte action
Final Answer/Add Decomp/Selt Answer
Add Decomp
Instr.
L question
d2
(ext)
de decomposition question
Q+
AD
CoreModel
dg
de
ft
extrocted foct
Select Action
Answer
final answer
Retrieve & Extract
Al oction space
Retrieve
c: oction
Q
Extract
C. corpus
Retriever
Extractor
f
Px poragrophs
d2
f
Figure 2: GEEK workflow: the core model, retriever and extractor collaborate solve complex questions progres- sively. (Left): In each iteration, based on the question state 2,, GEEK selects an action and calls the corresponding module to execute.
question state turn, until a final answer detailed procedure of action selection and execution. For action selection, Q, and A are fed into the core model with the instruction for action selection, and the model outputs an action code a. For the execution Add Decomp, Final Answer, and Self Answer, the core model outputs corresponding responses following different instructions. At last, for Retrieve & Extract, the retriever firstly retrieves several paragraphs P from the corpus external knowledge, and then the extractor answers the decomposition question d, based on P.
execution updates
derived. (Right):
4.1 Core model
we deploy the title retriever for a document-level retrieval, to shrink the context space to hdor: = 100 documents, where the context embeddings are built from each document's 's title and first paragraph. Then the paragraph retriever performs secondary search the paragraph level, outputting the top k matched paragraphs among the 100 documents.
The core model is pretrained LLM for sequence- to-sequence text generation (e.g.. Flan-T5 (Chung et al., 2022)), acting as the controller of GEEK. For each step t, the core model chooses action perform, among the action space A and condi- tioned on the current question state Q::
Core(Qu: A)
4.3 Extractor
Even though we retrieve A: paragraphs from the vast C, these retrieved paragraphs are still too long be input into the core model. Therefore, another specialized extractor is used condense the k para- graphs into concise fact sentences for d;
Besides choosing actions, the core model is also used to execute some types of actions, under action- specific prompts, such as generating dccomposition questions. Details are in Sec. 4.4.
4.2 Retriever
Extractor(di, P)
order to utilize external knowledge, we employ a neural retriever, DPR (Karpukhin et al., 2020), to retrieve paragraphs from vast volume of context C. Given the decomposition question de as query, the retriever returns top & relevant paragraphs:
We use FiD architecture (Izacard and Grave, 2020b) for the extractor. Instead of extracting facts locally from each of the paragraphs, FiD can perceive all the paragraphs simultaneously and generate more comprehensive results.
P. = {p1. P2. } Retr(di,
4.4 GEEK Pipeline and Action Space
where Retr stands for the retriever and each p. denotes paragraph. Due the enormous amount of context C, full-size retrieval is time costly. For efficiency, we use two nested DPR bi-encoder models (Karpukhin et al., 2020), namely the docu- ment retriever and the paragraph retriever. Firstly,
As shown in Fig. 2, GEEK iteratively selects and ecutes actions to solve implicit complex questions. The procedure involves neural inference where neu- ral networks generate text outputs that are less in- terpretable due to the black-box nature of neural

---

networks, as well as symbolic inference which fol- lows strict rules, Based on the question state 2, at each round, the core model decides whether a final answer can be made or more decomposition needs to be explored. If the latter, it also determines whether external knowledge should be retrieved, or the decomposition can be directly answered using the fact sentences far. The newly acquired facts are added the question state for the next iteration. Next, we introduce the details for each action:
pseudo answer fa from the 'AddDeoomp' ac- tion as a reference. We formulate the input to extractor with the prompt: 'Answer the question dy based on the context p; refer- ence but not necessarily correct answer is ft'. Therefore, the extractor knows what kind of information should be extracted among pos- sibly multiple aspects, avoiding correct but not relevant answers (e.g., for decomposition 'Who is xxx?', possible aspects include nation- ality, career, education, family, etc.).
FinalAnswer (core model) If enough back- ground knowledge is acquired and a final con- clusion can be drawn, the core model should output the final answer to q. For the real im- plementation, we let the model summarize the facts from previous steps as self-CoT, and then conclude the final answer (yes no).
SelfAnswer (core model) For some decom- position questions that are pure logical de- duction, or the required knowledge has been included in the question state already, no ex- ternal evidence is needed and hence retriever and extractor are not used. In this case, a self- answer prompt is used and the core model would answer de directly, outputting ft.
AddDecomp (core model) Based on current 2t, the core model generate next-step de- composition question de The generation conditioned on previous decompositions and facts. Hence the model could adjust the rea- soning strategy on-fly, and benefit from the gradually enhanced external knowledge.
4.5 Strategy Exploration
Considering that there often exist multiple possi- ble strategies to solve an identical question, we also extend the GEEK to explore a strategy space for different solutions. Specifically, in the step of AddDecomp, the core model can return multiple decomposition questions using beam search. For each different decomposition d,") a copy of cur- rent question state Q₂ is created, and updated by the new decomposition, forming 2(1) Then the copies carry on for the rest solving process independently. We emphasize that our method Strategy Exploration (SE) is different from Self- Consistency (Wang et al., 2022), which outputs multiple CoT solutions with the one-shot gen- eration, and then takes the majority. Under the scenario of GEEK with SE, the question branches into 1 different strategies every iteration, hence formulating latent solution tree. The diverged decomposition would lead to different retrieval results and generated facts, hence is an exploration of the strategy space. Due to computation constraints, we limit the expansion number to be at most 16 (i.e., expansion rate n = at each decomposition and for at most 2 iterations). The majority vote used derive the final answer.
To further improve the comprehensiveness of the strategy and avoid generating unsolv- able decomposition questions, we design a pre-answer trick. Instead of generating di only, we use explicit prompt to instruct the model generate all the remaining decom- positions with corresponding pseudo answers: (di, fe, f+11 ). Hence, the strategy leading by d, could be more coherent and solv- able. Note that the generated pre-answers are not necessarily correct, and all the redundant generations except de only serve as genera- tion auxiliaries, which will be removed before adding to the question state Q..
Retrieve & Extract (retriever) Once new decomposition is added, the core model would decide whether this decomposition question needs external knowledge to answer. yes, this action is executed and the retriever called to retrieve the top I most relevant para- graphs P; corresponding to the current decom- position d, Once P, is retrieved, the extractor model would read the l paragraphs and gen- crate concise fact f as the answer to d,
Case Study
In Fig. we show a full inference process of GEEK. The inference involves in total three iterations.
For the extractor, we also use the generated

---

Q: Did Alice's Adventures in Wonderland inspire Macbeth?
(Question state)
Question:
Questi on Stabe
Synthesis the information so far, what action should performed next? [A]: enough evidence to conclude final answer.
Did Alice's Adventures in Wonderland inspire Machbeth?
Previous
Sub-question: When was Alice's Adventures in Wonderland published?
BB: ask nore sub-questions for necessary knowledge.
Fact: 1865
(2) Sub-question: When was Machbeth published?
>>> Besides the current steps, what more sub-
Fact: 1623
questions need to asked? Show your steps in the format "(i) [Q] (sub-question) [A] (pseudo answer)"
(3) Sub qub--quetton:: Is 1865 before 1623?
Fact: The answer is: No.
Synthesis information far [A] final answer (B] ask nore sub- questions
CoreModel: (1) When Alice's Adventures Wonderland
published? [A] 1865 (2) What year was Maebeth written? [A] 1599 1865? No
[Q]
>>> what more sub-questions need to be asked?
>>> Does the sub-question "When was Alice's Adventures Wonderland published?" involves external evidence to answer?
CoreModel: (3) is 1865 before 1623 [A] No
external evidence is needed, it be answered without extra evidence
tools
>>> Does the sub cub-question "Is 1865 before [A]: call tools extra evidence
answered without
(Retrievrr Extractor calied)
1865
Using above sub-questions and facts, the sub-question using string logic: "Is 1865 before 1623?"
Synthesis the infornation far [A]: final answer B ask more sub- questions
CoreModel: The answer No.
>>> Besides the current steps, what more sub- questions
Synthesis information far final answer [B]: ask nore sub-
asked?
CoreModel: (2) [Q] When was Machbeth published? [A] 1603 (2)
questions
before [A]
>>> Conclude a final answer to the question "Did Alice's Adventures Wonderland inspire Nacbeth?" Use the fornat "Reasoning steps: [your steps] Therefore, the final answer is: [yes/no]"
Does the sub-question "When was Machbeth
published
AA call tools [B]: answered without
extra evidence
(tetriever and Extractor called) Extractor: 1623
Reasoning steps: Alice's Adventures Wonderland was published Macheth published Therefore, the final answer is: no.
CoreModel:
Figure 3: Full process of GEEK inference. For each round, the prompts are shown in gray, and the current question state also given to model as input. Model responses are shown in green and action selection is represented by red circle to save space. On the top right corner, the question state is listed, where the historical states of sub-question and fact are gradually added during the inference (best viewed in color and numerical marks).
order to save space, we omit the input question state for each step, but show it in the top right corner. Marked in colored numbers, the question state is gradually enriched as the decomposition questions and corresponding facts are added. Also, for each action selection step, circle the core model's choice. Some repeated prompts are also abbreviated, due to the page limitation.
also dynamically adjusted as more facts are ac- quired (e.g.. 'Is 1599 before 1865? + 'Is 1865 before 1623?'). For the last logical sub-question that requires no external knowledge, the core model chooses to self-answer it without calling retriever and extractor. When there is enough information, the final answer 'no" is given with reasons.
Experiment
As can be observed in the example, the GEEK decomposes the original question into three sub- questions, and takes the strategy of 'temporal con- fliction' to solve the problem. For sub-questions '(1)' and '(2)', external knowledge is retrieved and the original pseudo answers are verified and cor- rected (e.g., '1599' + $1623'). The strategy is
In this section, compare GEEK with previous baselines and conduct ablation studies analyze the contributions of specific modules.
6.1 Dataset and Preprocessing
We use the StrategyQA dataset (Geva et al., 2021) to evaluate our method. The dataset consists

---

Method
Backbone
Retrieve
Specification
SQA
ChatGPT (Qin et al., 2023)
GPT-3.5 (175B)
X
Without CoT
59.2
ChatGPT (Qin et al., 2023)
GPT-3.5 (175B)
X
CoT
62.5
FaithfulCoT (Lyu al., 2023)
code-davinci-O00 (175B)
X
73.2
(Xie et al., 2023)
code-davinci-002 (175B)
X
77.2
(Lazaridou al., 2022)
Gopher (280B)
66.2
Visconde (Pereira al., 2023)
text-davinci-002 (175B)
CoT CoT
69.43
RR (He et al., 2022)
text-davinci-002 (175B)
77.73
PaLM (Chowdhery al., 2022)
PaLM (540B)
73,9
PaLM (Anil et al., 2023)
PaLM (540B)
CoT + SC
81.6
Pa (Anil et al., 2023)
PaLM2 (340B)
X
90.4
GEEK (ours)
Flan-T5 (11B)
CoT 75.98
GEEK (ours)
Flan-TP (11B)
CoT+SE 78.17
Table Experiment results on strategy dataset. GEEK achieves the SOTA accuracy for LLMs in ~-0B scale, and surpasses all the previous methods with backbone under 300B scale, using only 6% parameters less.
2061 samples in the train set and 229 in the dev set. Another 490 samples without labels are provided for the test set, and results for that be uploaded their website for evaluation
2018) as backbone. By default, k = 10 paragraphs will be retrieved during GEEK inference. The ex- tractor is FiD (Izacard and Grave, 2020b) with Flan-T5-3B the backbone. During training, the extractor fed golden paragraphs and remaining retrieval paragraphs, to satisfy the fixed number k. For the core model, we adopt Flan-T5- 11B (Chung et al., 2022; Raffeel et al., 2020). The model is trained for multiple tasks including tion selection and execution of three actions, described Fig. 2. The tasks are trained paral- lel, with onput-output pairs built from human an- notations and in the same instruction format as Fig. 3. We train the model on V100 GPUs. Due to the LLMs' out-of-memory problems, the deep- speed flatform is used (Rasley al., 2020). During inference, utilize the accelerate package with offloading for acceleration (Sylvain al., 2022).
For train and dev samples, each question is provided a human- annotated strategy DD the form of decomposition questions. Golden supporting paragraphs for each d are also given, which are from 36.6M Wikipedia processed corpus. In addition, numan-annotated background facts F are also provided.
Nonetheless, the provided background facts not strictly mapped with the decomposition, either in the aspect of total number or sequential order. Additionally, over 25% human annotated decom- position questions refer to previous i-th decompo- sition by symbol '#i.. We find that these symbols cannot be simply filled by decomposition' 's corre- sponding answer, because some ##'' may refer to an entity in the decomposition question text. For the above reasons,
GPT4 (OpenAT, 2023) refine the annotations. We provide the question together with the final answer, golden facts and decomposition questions to GPT4, and prompt it to fill the #i's in annotations. Meanwhile, we also ask it to give a concise answer for each decomposition, according to the golden facts and the final answer. Unless specified, all the experiment results below are from the GPT4 processed version.
6.3 Comparison with other Baselines
In this section, we compare the proposed GEEK with other baselines on the strategy QA dataset. As shown Tab. 1, GEEK yields 78. 17% accuracy on the StrategyQA dataset, with a much smaller model size (11B) than previous baselines. Our method sets new SOTA for the 10B LLMs, and also is the second best among all existing methods except PaLM. Although GEEK is finetuned with supervision, the accuracy of GEEK is still consid- erably high considering its size in comparison with other baselines. also worth noting that due the indirection using LLM APIs, finetuning or adaption for special domain tasks not easy. For
6.2 Detailed Settings
For retriever, vanilla DPR (Karpukhin et al., 2020) is used with BERT-base-uncased (Devlin et al.,

---

generated decomposition-fact pairs, and let the model choose which more informative and faith- fully correct. The results are listed in Tab. 3. turns out that 62.45% of the GEET-generated decomposition-faan pairs are preferred.
De
RE
SE
Acc
Zero-shot
x
X
X
62.01
CoT
X
X
X
70.74
+De
V
X
71.50
+RE
J
75.98
Full
78.17
Conclusion
Table 2: Ablation study results. The three columns de- note whether an action is performed under GEEK. ('De": 'AddDecompp, 'RE": 'Retrieve and Extract', 'SE": 'Strat- egy Exploration')
We present GEEK, pipeline to progressively ex- cavate external knowledge for boosting LLM's ca- pability in solving open-domain multi-step implicit questions. Interactively, GEEK decomposes the question into explicit sub-questions to retrieve ex- ternal knowledge, and the accumulated knowledge enlightens the model to form a better strategy in turn. The proposed method also provides explain- ability by showing the full reasoning process, sup- ported by retrieved evidence. By using SE, GEEK also explore the strategy space and try different approaches to solve the complex question, which increases the performance further. Experiment re- sults justify our design. With GEEK, 78.17% accu- racy is achieved using less than 6% size of the com- petitors, refreshing the SOTA accuracy for 10B LLMs. As an alternative to the paradigm of scaling for larger models and more pretraining data, we hope this research could inspire more future works to investigate how organically excavate external knowledge and progressively formulate strategies for solving open-domain implicit questions.
Human
Equal
GEEK
ChatGPT 13.54%
24.02%
62.45%
Table 3: ChatGPT Assessment of prediction results.
example, only Visconde (Pereira et al., 2023) and RR (He et al., 2022) successfully leverage exter- nal knowledge, but with a relatively less capable backbone text-davinci-002. However, the external knowledge is proved to benefit the task effectively. and RR achieves SOTA performance at its time. Specifically, while the initially finetuned GEEK yields 75.9 58.9 accuracy, by using SE, the accuracy is improved significantly, to 78. 17%. This process requires no retraining yet can boost final accuracy, where GEEK explores the strategy space and tries to solve the question via multiple paths.
6.4 Ablation Study
Limitations
We also analyze the contribution of the different components in GEEK. Results are shown in Tab. The line of 'CoT" denotes that the core model di- rectly answers the question as in the 'FinalAnswer' action, following the CoT approach (Wei et al., 2022a) without iterative reasoning and knowledge retrieval. After finetuning, the accuracy is 70.74% accuracy, 8.73% higher than zero-shot Flan-T5 but 7.73% lower than the full version of GEEK. We find that performing the action "Extritve=Extratt could efficiently increase the accuracy to 75. 75.98% justifying the motivation of leveraging external knowledge for solving open-domain questions.
Albeit achieving outstanding overall accuracy with a significantly smaller model size, the GEEK is not without limitations. First of all, as long as neural reasoning is involved, the hallucination problem is inevitable in theory, due to the black-box nature of neural networks. We want stress that by us- ing a retriever and an extractor, the hallucination problem can be alleviated from factual references, but not completely avoided. Secondly, the logic of GEEK is not guaranteed to be correct. is possible that GEEK gives the correct answer but wrong solv- ing steps, or correct intermediate steps but wrong final answer. Lastly, due to the scarcity of public datasets like Strategy for open-domain complex question answering, is difficult for to research this problem under more datasets and different task settings. We expect future works that may come up with more suitable datasets like StrategyQA.
6.5 ChatGPT Assessment
To further evaluate the quality of the strategy gen- erated by GEEK, we also leverage the GPT4 to simulate human assessment. Specifically, we show it the numan-annotaie decomposition ques- tions and corresponding facts, as well as the GEEK-

---

References
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin. Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2355.0003.
Uri Katz, Mor Geva, and Jonathan Berant. 2022. In- ferring implicit relations in complex questions with language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2548 2566.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:220..03311.
Angeliki Lazaridou, Elena Gribovskaya, Wojeiech Stokowiec, and Nikolai Grigorev. 2022. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115
Hyung Won Chung, Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling cntructioo-finetunee language models. arXiv preprint arXiv:2210.1144
Shaobo Li, Xiaoguang Li, Lifeng Shang, Xin Jiang, Qun Liu, Chengjie Sun, Zhenzhou Ji, and Bingquan Liu. 2021. Hopretriever: Retrieve hops over wikipedia to answer complex questions. Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 13279-13287
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova, 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arviv preprint 1rXiv:1810.04800
Zhengzhong Liang. Tushar Khot, Steven Bethard, Mi- hai Surdeanu, and Ashish Sabharwal. 2022. Better retrieval may not lead to better question answering. arXiv preprint arXiv:2200 2205.0385..
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot,
Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346 361.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of- thought reasoning. arXiv preprint arXiv:2301.133: 13379.
Tim Hartill, Diana Benavides-Praise Michael Witbrock, and Patricia J Riddle. 2023. Answering unseen ques- tions with smaller language), (oongge)/mmmmee moodls using ratio- nale generation and dense retrieval. arXiv preprint arXiv:2308.04711,
Fedor Moiseev, 7he Dong, Enrique Alfonseca, and Mar- Jaggi. 2022. Skill: Structured knowledge infusion for large language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1581-1588.
Hangfeng He, Hongming Zhang, and Dan Roth. 2022. Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301.00 .03303.
OpenAL. 2023. Gpt-4 technical report. Computation and Language, arXiv:2303.08774.
Gautier Tzacard and Edouard Grave. 2020a. Lever- aging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.
Jayr Pereira, Robson Fidalgo. Roberto Lotufo, and Ro- drigo Nogueira. 2023. Visconde: Multi-document qa with gpt-3 and neural reranking. In European Conference Information Retrieval, pages 534-543. Springer.
Gautier Izacard and Edouard Grave. 2020b. Lever- aging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt purpose natural language process- ing task solver? arXiv preprint arXiv:2302.0644
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brah- man, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. Maicutic prompting: Logically consistent reasoning with recursive explanations. arXiv preprini arXiv:2205.11822
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21 21(1):5485-5551
Minki Kang. Seanie Lee, Jinheon Back, Kenji Kawaguchi, and Sung Ju Hwang. 2023. Knowledge- augmented reasoning distillation for small language models in Knowledge-intensive tasks, arXiv preprint arXiv:2305. 18395.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimiza- tions enable training deep learning models with over

---

100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 3505-3506.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao. Zhao, Min- Yen Kan, Junxian He, and Qizhe Xie. 2023, De- composition enhances reasoning via self-evaluation guided decoding. arXiv preprint arXiv:2305.0060
Danilo Ribeiro, Shen Wang, Xiaofei Ma, Rui Dong, Xi- aokai Wei, Henry Zhu, Xinchi Chen, Zhiheng Huang. Peng Xu, Andrew Arnold, al. 2022. Entaillmett tree explanations via iterative retrieval- -aarreeallgenraationn generation rea- soner. arXiv preprini arXiv:2205.090 09224.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning. 2018. Hotpolqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the pa- rameters of language model? arXiv preprint arXiv:2002.089100
Shunyu Yao. Jeffrey Zhao. Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Syn- ergizing reasoning and acting in language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline Sancock-Beauliee Mike Gatford, et 1995. Okapi at tree-3. Nist Special Publication Sp. 109:109.
Yunchang Zhu, Liang Pang, Yanyan Lan, Huawei Shen, and Xueqi Cheng. 2021. Adaptive information seek- ing for open-domain question answering. arXiv preprint prXhv:2109.0677
Gugger Sylvain, Debut Lysandre, Wolf Thomas, Schmid Philipp, Mueller Zachary, and Mangrulkar Sourab. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. https:// github. com/huggingface/// accaaceee
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. Alpaca: strong, replicable instruction-following model. Stan- ford Center for Research Foundation Models. https://orfmm stanford. 2///2222///331//apppaa html, 3(6).7.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1323 139771
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleav- ing retrieval with chain-of-thought reasoning for sewlledge-intensiie multi-step questions. arXiv preprint 12.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning language models. arXiv preprint 22Xi:22033.111
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le. Denny Zhou, al. 2022a. Chain-of thothtttt prompting elicits soning in large language models. Advances Neural Information Processing Systems, 35:24824-24837.
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, al. 2022b. Chain-of-thought prompting elicits rea- soning in large language models. Advances Newral Information Processing Systems.

---

Appendix
LLaMA-13B backbone yields similar accuracy Flan-T3-11B demonstrating the generality of our GEEK pipeline. However, LLaMA trained with- out instruction tuning, and both LLaMA-7B and LLaMA-138 perform slightly worse than F-an-T5- 1IB. This observation suggests that instruction tun- ing is helpful for the task of StrategyQA, but a larger model (e.g.. LLaMA-13b) can reduce the gap.
A Training Details
For training the core model, we use the refined annotation of StrategyQA dataset. The original StrategyQA dataset provides human-annotated de- composition questions, fact sentences, and support- ing paragraphs from Wikipedia. The decompo- sition questions break the original open-domain complex question into several simple and explicit sub-questions. The fact sentences contain evidence and background knowledge for solving the main question. The supporting paragraphs are where the fact sentences are grounded, and are one-to- one matched with the corresponding fact sentences. Nonetheless, the fact sentences are not one-to to-one mapped to each decomposition question. There- fore, we use GPT4 to refine the annotation, ex- tracting a concise answer for each decomposition question from the set of auman-annotatie fact sen- tences. Explicit instruction is given to GPT4 to force the model use information that faithful to the human-annotated fact sentences only.
D Prompt Examples
In this section, we show all the prompts used in GEEK, please also refer to Sec. 4.4 for more details of the actions. The terms such (Quetioo_state) represents the placeholder be substitute by corresponding
values.
System Prompt (at the beginning every in- put):
"Try.to..solve.a.mulr -tsp............
open - desenddomiinn_utttti </s >(Question_state)
After that, the core model is trained via super- vised fine-tuning- Specifically, we use ground-truth annotations (decomposition, paragraphs, and fact sentences) to build full-solving process for each question. The ground-truth action to be performed also depends the process. For example, if there are still more ground-truth decomposition ques- tions, the model should select 'AddDecomp'. And if all decomposition questions have been visited, the model should select 'FinalAnswer'. By doing so, we build multiple input-output pairs simulating each timestep of the solving process, and use these pairs to train the core model in parallel.
ActionSelection Prompt1 (begin of each round):
"Sytthesissthee.iiffrrtttioooo
far hwt..
-pperformed_thheeeexxtt_
]:.enoughh-evidencee.coo
conclude.a.finaalaasveee... nbB questions__or..ncessee knowledge."
If final answer (action [A]):
"Comclude.aafinnalaasvsee...
B More Comparison with Other Baselines
the..puestiooo..C...buue...tt
Besides the results shown in Tab. 1, we also list more comparison results with other baseline meth- ods, whose backbone models ours. As shown in Tab. 4, our method signifi- cantly surpass the other baselines, with improve- ment larger than 7.62%.
format_.iReasonnnn__stpsss [your_reeasoming_sttppp]]
similar in size
Therefore,
answer myswer_.s:._ceesso]]]]
If add decomp (action [B]):
Results with Other Backbones
"essidesstheeccurennttttss
what_moree.su--quusiions
We also implement GEEK with other backbone models, verify the generality of our method. In this section, we select the similar-scale ILaMA models for comparison. As shown Tab. 5, GEEK performs well with all the backbone models. The
need_to.bbe.ssked2_.hhoo
your_steps..nntthe.formes
V=(i)_[[Ql-ccub--uusstiiin (pp(sueudo__answerr

---

Method
Model size
Acc
UL2 (Qin et al., 2023)
20B
59.0
Stable INT8 (Hartill et al., 2023)
13B
61.7
GR+RATD (Hartill al., 2023)
440M
64.2
KARD (Kang al., 2023)
3B
70.55
GEEK (ours)
11B
78.17
Table 4: More comparison with other basclines.
E Error Analysis of GEEK
LLM
Acc
LLaMA-7B
74.67
We manually analyzed all the error samples in the dev set. The reasons for wrong predictions are categorized into 4 types listed below (with actual examples):
LLaMA-13B
77.73
Wan-TT5-11
78.17
Table 5: More backbone results.
Unreasonable decomposition: the predicted decomposition does not effectively lead a valid solution to the original answer. For ex- ample:
ActionSelection Prompt2 (when new decomp is added):
"Does_ "Dess_the_.uu--uusttioooo--eee
Question: "Would the author of Little Women have remembered the ratification of the 13th Amendment?"
oxidence_.to.answerr2..tLLL
.external.eev.dence..s..
Ground truth: "(1) When was the 13th Amendment ratified? (2) Who wrote Lit- te Women? [Louisa May Alcott] (3) What years was Louisa May Alcott alive? [1832- 1888] (4) Did the ratification of the 13th Amendment occur sometime during 1832-18887 [Final answer: yes] Prediction: "(1) When the 13th Amendment ratified? (2) When was Louisa May Alcott born? [1832] (3) 1865 before 1832?" [Final answer: no]
mecded_.ccal..ttools...1111
.it.cam.bee.sfeely.asswer.
.ly_Logicaaliinferencee without..extra.evvinn
If call tools (action [A]):
# retriever input
"Question: _uu0)) [Decomp)"
extractor input "Baeee_on_.the_folloviin context,..anserr.tten question: reference..utt.not_
Wrong action selection: the model selects the wrong action to be performed (e.g., con- clude final answer too early, incorrectly call retriever, or attempt self-answer)
""
necesseilly.correet.nnww
Incorrect facts: put incorrect facts (either irrelavant or factu- ally incorrect). For example:
Retriever and extractor out-
_is::\V(ppseddo_aaswer\\
c/seContext:
Decomposition question: "What are the numbers that are used in the scoring sys- tem in table tennis?"
If self answer (action [B]):
GT: "11 and 21" (for old rules).
logic.to..nswer_theeee
Pred: "15 points, 30 points, and 40 points".
(uestion:_

---

Logical deduction error: the model gives the wrong final answer from correct decomposi- tions and facts. For example:
Question: "Can the Swiss Guard fill the Virginia General Assembly chairs?"
Facts: "There are 140 seats in the Vir- ginia General Assembly. The Swiss Guard has total of 134 members."
-T answer: "No"
Pred answer: "Yes"
Based on the above categories, we also statistic the proportion of errors. The results are shown in Tab. 6. Most of the errors due to bad decompo- sition generation and incorrect facts. Because of the nature implicit QA, generating good decom- position questions is challenging. By observing the error cases, we find that logical reasoning and background knowledge are required and critical to generate high-quality decompositions. Therefore, we hypothesize that GEEK could benefit from larger backbone LLM, which is believed to have more knowledge absorbed and higher reasoning ability. This is also partially verified by the results in Tab. and Tab.
Error type
Percentage
Unreasonable Decomposition
40%
Wrong action selection
8%
For the factual errors generated facts, as also mentioned the section 'Limitation', , the extrac- tor's neural processing mechanism makes the hal- lucination problem inevitable. Based on the obser- vation of error cases, we find that two main reasons result in incorrect facts: (1) the retriever fails to find relevant paragraphs corresponding the de- composition question, and (2) the extractor outputs summarized fact sentences that are not faithful to the paragraph. For (1), a more powerful retriever (e.g.. search engine) or more affluent corpus be- yond Wikipedia could help. For (2), faithful QA techniques, such as changing the generative task into an extractive task, might be remedy. How- ever, constraints by the computation resources, and also considering that this rescarch mainly focuses on implicit question answering, we leave this prob- lem for future works.
Incorrect facts Logical deduction error
54%
20%
Table 6: Error cases and proportion.


Reviews:
[{"review_id": "jt0q5efZYK", "reviewer": "Reviewer_Ab27", "summary": "", "questions": "Q1. How the core module is fine-tuned exactly to decompose question and decide when a final answer is reached?\n\nQ2. What are the technical challenges in implementing the core module and how are they addresses if any?\n\nQ3. How does multi-step iterative decomposition improves model performance? What is the baseline performance if the model only performs a single retrieval step/reasoning?\n\nQ4. The proposed approach is tailored for implicit QA; is it also applicable to hoptopQA?\n\nQ5. What's the role of GPT4 dataset refinement on model performance? What's the baseline performance without GPT4 refinement? Is there any plan to release the refined dataset?\n\nQ6. How does parallel \"strategy exploration\" differ from tree-of-thought [1].", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-motivated and the use case demonstrations for implicit question answering is interesting and insightful.\n\n2. The paper proposes a comprehensive multi-step question answering framework. The three main components of the system are well-designed: action space of the core module can decide decomposition/halting/self-answer, retriever and extractor can retrieve and condense external knowledge, parallel reasoning paths can improve the quality/robustness of the results.\n\n3. Ablation studies validate the effectiveness question decomposition and knowledge retrieval components.", "reasons_to_reject": "Overall, I liked the idea and proposed framework. My main concern is in the lack of technical detail, sufficient experimental validation, and potential challenges with reproducing the results.\n\n1. *Lack of technical detail in how the core module is fine-tuned.* The proposed approach is mostly described at a conceptual level and lacks in-depth technical detail regarding how the core module is fine-tuned. This is of particular concern given the sophistication of the proposed system and the importance of the core module in performing question decomposition, output final answer, and deciding tool usage. The absence of such technical details could hinder the reproducibility of the approach, especially since the source code is not available.\n\n2. *Insufficient validation of iterative problem decomposition.* In the related work, the paper argues the main contribution over other retrieval-based multi-step QA methods is on the application of implicit questions and the need for multi-step progressive reasoning or tool use. Although ablation study confirms the effectiveness of question decomposition in general, it lacks analysis on the number of iterative steps performed and how it improves model performance. As the model has only been validated on a single dataset, StrategyQA, it remains unclear how the proposed approach would fare on “simpler” cases such as hotpotQA.\n\n3. *GPT4 dataset refinement confounds the contribution of this paper.* The proposed approach is fine-tuned on a version of the StrategyQA dataset that is refined by GPT4. This raises the possibility that the fine-tuned model may have learned or distilled behaviors from GPT4, behaviors that might have been difficult to acquire from the original dataset. While distillation is an accepted technical method, the paper does not clarify the degree to which GPT4 refinement contributes to improving model performance. Consequently, this creates an unfair comparison with other models that are not fine-tuned on GPT4 outputs and depend on prompting.\n\n", "questions_to_authors": "Q1. How the core module is fine-tuned exactly to decompose question and decide when a final answer is reached?\n\nQ2. What are the technical challenges in implementing the core module and how are they addresses if any?\n\nQ3. How does multi-step iterative decomposition improves model performance? What is the baseline performance if the model only performs a single retrieval step/reasoning?\n\nQ4. The proposed approach is tailored for implicit QA; is it also applicable to hoptopQA?\n\nQ5. What's the role of GPT4 dataset refinement on model performance? What's the baseline performance without GPT4 refinement? Is there any plan to release the refined dataset?\n\nQ6. How does parallel \"strategy exploration\" differ from tree-of-thought [1].", "paper_topic_and_main_contributions": "The paper introduces a framework for open-domain implicit question answering that requires multi-step reasoning and access to external knowledge.\nMotivated by LLMs’ limitations in “uncovered or out-of-date parametric knowledge”, the proposed approach aims to progressively break down complex problems and retrieve/extract external knowledge to answer implicit complex questions.\n\nThe main contributions include: (1) a Gradually Excavating External Knowledge (GEEK) framework that uses three modules—core, retriever and extractor—to perform multi-step reasoning with access to tools. The core module determines question decomposition and the halting condition. The retriever module performs retriever at the title and paragraph level, and the extractor module condenses retrieved content into concise sentences. (2) Employing parallel strategies to solve a problem by simultaneously exploring multiple decomposition trajectories. (3) A pre-answer trick that uses mock answers output from the model to constrain the pipeline search. (4) Evaluation on the StrategyQA dataset demonstrate competitive performance 78.17% accuracy using a much smaller model (FLAN-T4-11B). (5) Ablation studies demonstrate the effectiveness of knowledge retrieval.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Yao, Shunyu, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. \"Tree of thoughts: Deliberate problem solving with large language models.\" arXiv preprint arXiv:2305.10601 (2023)."}}, {"review_id": "nmgYZ8VoJt", "reviewer": "Reviewer_DMUD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The model achieves the SOTA performance with a much smaller parameter size.\n2. The authors propose a gradual knowledge excavation framework for open-domain complex question answering.\n3. The authors announce they are the first to propose a novel pipeline to solve open-domain complex questions by progressively acquiring external knowledge and adjusting its strategy.", "reasons_to_reject": "1. Readability. Many typos and Method related parts such as Sections 4 and 5 can be improved for easier understanding.\n2. More experiments on different open domain question answering datasets and more relevant model comparisons are needed.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose to extract knowledge bases step by step to solve the open-domain complex question-answering task instead of extracting all knowledge context at one time. They decompose questions into sub-questions for extracting knowledge. In addition, they use FLAN-T5 as a transaction liked model to determine when to stop extracting knowledge using sub-questions and answer questions. The experiments are only conducted on StrategyQA and achieve 78.17% accuracy compared to the previous SOTA performance, which is 77.73%. But the parameter size of the proposed model is 6% of the previous SOTA model.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Trivedi, Harsh, et al. \"Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions.\" arXiv preprint arXiv:2212.10509 (2022). A similar work. Please compare with this paper, this paper also achieves high performance with small models using similar pipelines proposed in the submission.", "Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\nPage.2 LINE 100, right: 77.73% -->78.17%\nPage.3 LINE 188, right: eternal --> external\nPage.3 LINE 195, right: q is open-main -->q is open-domain\nPage.4 LINE 210, left: Details in Sec. ??\nThere are many other typos, a careful proof reading is suggested."}}, {"review_id": "LMJqYycKpb", "reviewer": "Reviewer_2gMb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The pipeline proposed by the authors for the task of implicit question answering appears to be relatively straightforward to implement and shows promising results when incorporating external knowledge.\n- The method description is detailed and the writing is clear.", "reasons_to_reject": "I believe this work is of high value, as it tackles the specific task of complex problem reasoning. However, I have some concerns:\n\n-  Currently, tools like langchain, which integrates ReaAct, can already build tool learning pipelines by calling interfaces. These models have shown promising results on complex tasks that require external knowledge. Considering the availability of such tools, what are the advantages of this research?\n- The experiments in this paper were only conducted on the strategyQA dataset, including the main experiment and the ablation stud of CoT/SE. There is a lack of in-depth analysis of the reasons for errors and error statistics, making it difficult to assess the actual effectiveness of the pipeline.\n- There is a lack of actual examples and prompt demonstrations.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a pipeline for implicit question answering in the open domain. The pipeline is built upon the core module of LLM, which serves as the decision-making center to determine and execute actions, decompose questions, and gradually obtain the final answer. The authors conducted experiments on the strategyQA dataset and demonstrated the effectiveness of this pipeline.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YUuAdmLs0H", "reviewer": "Reviewer_4HSn", "summary": "", "questions": "A.Have the authors considered conducting some generalization experiments on other tasks? For example, the CommonsenseQA, this will demonstrate the superiority of the model.\nB.Can the authors conduct some error analysis?\nC.It seems necessary to present a comparison with models of the same or smaller scale. Why didn't the author do this?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. This paper carefully designed a pipeline model that gradually excavates knowledge from an external knowledge base to solve complex problems. It solves the problem of one-time retrieval by decomposing problems and iteratively retrieval.\nB. This paper designed a rigorous and complex reasoning process, utilizing the existing excellent retrieval algorithm (DPR) and answer extractor (FID) as auxiliary tools to obtain the retrieved facts. By fine-tuning Flan-t5-11B to control the next action selection, decompose complex problems into sub problems, and generate answers through internal knowledge directly. It solve the hallucination problem of LLMs to some extent.", "reasons_to_reject": "A. It is not sufficient to conduct experiments solely on StrategiyQA, which reduces the applicability and makes it hard to prove the effectiveness of the model.\nB. The paper lacks necessary error analysis.\nC. The author did not provide a detailed description of how to fine tune the core model for action selection, and how to determine whether the model can do self-answer.\nD. The author used GPT4 to re-annotate the data, which increased costs and appeared to be an unfair comparison.", "questions_to_authors": "A.Have the authors considered conducting some generalization experiments on other tasks? For example, the CommonsenseQA, this will demonstrate the superiority of the model.\nB.Can the authors conduct some error analysis?\nC.It seems necessary to present a comparison with models of the same or smaller scale. Why didn't the author do this?", "paper_topic_and_main_contributions": "This work proposes a gradual knowledge excavation framework called GEEK for open-domain complex question answering, where LLMs iteratively and actively acquire extrinsic information, then reason based on acquired historical knowledge. GEEK consists of three modules, core model, retriever and extractor. The core model handles logical reasoning and selects an action to perform at each time step, planning the solving strategy purposefully. The retrievers allocate relevant context paragraphs from the external corpus (e.g., Wikipedia) to provide trustworthy knowledge, and the extractor condenses the textual knowledge into brief fact sentences. The experimental results on the challenging StrategyQA benchmark surpass vanilla LLMs such as ChatGPT with 94% parameters less, refreshing the new SOTA for LLMs under ∼300B scale.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 192, The formula is difficult to understand\nLine 344 'GEED' should be 'GEEK'\nLine 210, reference error\nLines 679-680, no appendix exists, should be deleted"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces GEEK, a method that employs multi-step reasoning and retrieval from external knowledge sources to answer implicit questions. Such questions typically pose challenges for LLMs due to outdated parametric knowledge. GEEK incorporates both question decomposition and a halting condition. On StrategyQA, GEEK achieves a new state of the art of 78.17%, up from the previous 77.73%, while using slightly fewer parameters. The approach is well-motivated and intriguing. Given the rising prominence of LLMs, this paper could be of significant value to the community. We strongly encourage the authors to address and refine the areas highlighted by the reviewers.
Decision: Accept-Findings