IAG: Induction-Augmented Generation Framework for Answering Reasoning Questions
Authors: Zhebin Zhang
Xinyu Zhang
Yuanhang Ren
Saijiang Shi
Meng Han
Yongkang Wu
Ruofei Lai
Zhao Cao
Venue: EMNLP 2023 Main (2023)
Keywords: Open-domain question answering
Inductive reasoning
Prompting

Abstract: Retrieval-Augmented Generation (RAG), by incorporating external knowledge with parametric memory of language models, has become the state-of-the-art architecture for open-domain QA tasks. However, common knowledge bases are inherently constrained by limited coverage and noisy information, making retrieval-based approaches inadequate to answer implicit reasoning questions. In this paper, we propose an Induction-Augmented Generation (IAG) framework that utilizes inductive knowledge along with the retrieved documents for implicit reasoning. We leverage large language models (LLMs) for deriving such knowledge via a novel prompting method based on inductive reasoning patterns. On top of this, we implement two versions of IAG named IAG-GPT and IAG-Student, respectively. IAG-GPT directly utilizes the knowledge generated by GPT-3 for answer prediction, while IAG-Student gets rid of dependencies on GPT service at inference time by incorporating a student inductor model. The inductor is firstly trained via knowledge distillation and further optimized by back-propagating the generator feedback via differentiable beam scores. Experimental results show that IAG outperforms RAG baselines as well as ChatGPT on two Open-Domain QA tasks. Notably, our best models have won the first place in the official leaderboards of CSQA2.0 (since Nov 1, 2022) and StrategyQA (since Jan 8, 2023).

IAG: Induction-Augmennted Generation Framework for Answering Reasoning Questions
Zhebin Zhang, Xinyu Zhang; Yuanhang Ren, Saijiang Shi,
Meng Han, Yongkang Wu, Ruofei Lai, Zhao Cao
Huawei Poisson Lab, China
(zhangzhebin2, zhangxinyu35, renyuanhang Jehuawei com
Abstract
extensively studied within the scope of informa- tion retrieval (Karpukhin et al., 2020; Das et al., 2019; Sachan et al., 2021), where access to external knowledge sources such as web pages or knowl- edge bases is required. Another line of research exploits large language models (LLMs) such as GPT-3 and PaLM the knowledge source (Petroni et al., 2019; Liu et al., 2022b) and develops various prompting methods to elicit knowledge from LLMs that implicitly encoded in the parameters. How- ever, to answer reasoning questions, i.e., questions that demand some degree reasoning ability, ei- ther retrieval-based or prompting-bases approaches suffer from their intrinsic limitations.
Cetrieval-Augmenne Generation (RAG), by incorporating external knowledge with para- metric memory of language models, has be-
come the state-of-the-arr architecture for open- domain QA tasks. However, common knowl- edge bases inherently constrained by lim- ited coverage and noisy information, making retrieval-hase approaches inadequate an- swer implicit reasoning questions. In this paper, we propose an Induetion-Augmennte Generation (LAG) framework that utilizes in- ductive knowledge along with the retrieved documents for implicit reasoning. We lever- age large language models (LLMs) for deriy- ing such knowledge via a novel prompting method based on inductive reasoning patterns. On top of this, implement versions of IAG named IAG-GPT and IAG-Student, respec- tively. IAG-GPT directly utilizes the knowl- edge generated by GPT-3 for answer prediction, while TAG-Student gets rid of dependencies on GPT service at inference time by incorporating a student inductor model. The inductor is firstly trained via knowledge distillation and further optimized by back-propagating the generator feedback via differentiable beam scores. Ex- perimental results show that IAG outperforms RAG baselines well as ChatGPT on two Open-Domain QA tasks. Notably, our best models have won the first place the offi cial leaderboards of CSQA2.0 (since Nov 1, 2022) and StrategyQA (since Jan 8, 2023).
On the one hand, although RAG (Lewis et al.. 2020) has become the SOTA architecture for ODQA, documents retrieved from common knowl- edge bases generally suffer from the limitations of constrained coverage and noisy information, es- pecially for implicit reasoning questions whose answers are not well documented. For example. it's trivial for an average child to answer questions such as "can you catch a jellyfish in the dumpster?". However, it's unlikely to find the answer directly from the web or books. As shown in Figure 1, the retrieved documents can hardly answer the ques- tion in such cases. Hence, relying entirely on in- formation retrieval insufficient solve implicit reasoning questions.
On the other hand, prompting- based methods can exploit the considerable amount of knowledge encoded in the parameters of LLMs for QA tasks (Wang et al., 2022; Wei et al., 2022; Chowdhery et al., 2022; Brown et al., 2020). But the problem of hallucination (i.e., generating natural language statements that are factually incorrect) imposes lim- itations LLMs in terms of factuality and credibil- ity. To better control the knowledge elicited from LLMs, various prompting methods such as chain- of thougghh (CoT) (Wei et al., 2022) have been pro- posed by constructing intermediate reasoning steps
1 Introduction
Open- -pen-Domain Question Answering (ODQA) (Zhu al., 2021) has attracted increasing research at- tention. Compared with the closed-domain set- ting, techniques developed upon ODQA models can empower search engines with the ability to respond to wider range of user queries. As typical inoeledge-intensivee task, ODQA has been
"These authors contributed equally to this work. *Corresponding author.

---

Disckamerr pelyfish you ouuld' your dumpoer... Retrieved Documents
Can you citch jelyhis dumpstert
Generator
Question
(Anzigg)
Crabs and strizpps
Answer
Jelyfih, cmbbs and shrimps are-rruatt enmals you aquatic animalsin the dumpsten
Jllyyfihh
(Gesralinnic))
Aquatic
Inductive Knowledge
Inductur
Figure 1: Overview of our IAG framework. The inductor provides inductive knowledge for predicting the answer when the retrieved documents are less informative.
until arriving the conclusion. However, capa- bility of the an LLM is intrinsically constrained by its parameter size, making it unable respond correctly to domain-specific questions beyond the scope of its training corpus.
The contributions of this paper include: 1) an inductive prompting method which improves the factuality of knowledge elicited from LLMs by constructing reasoning path via inductive reason- ing: 2) an IAG-GPT implementation that improves over strong baseline models and ChatGPT by leveraging the knowledge elicited from GPT-3 as auxiliary supporting evidence for the generator; 3) TAILBACK optimization algorithm to train the in- ductor which allows IAG-Student to outperform RAG baselines under small model size.
In view of the above challenges, it requires a new paradigm for building models applicable to reason- ing QA. To this end, we combine the advantages of these two kinds of methods and propose IAG, Induction-Aummenice Generation framework for answering implicit reasoning questions. IAG en- hances the conventional RAG with an inductor that generates inductive knowledge w.r.t each question. To derive such knowledge, we propose a novel prompting method, which is intuitively inspired by the cognitive functions of inductive reasoning, to elicit inductive knowledge from an LLM (i.c., GPT-3). Our first implementation of IAG, which is termed IAG-GPT, directly leverages the knowledge statements sampled from GPT-3 as evidence along- side the retrieved documents to feed into the an- swer generator. We show that IAG-GPT improves over SOTA models on multiple ODQA datasets and has won the first place in the official leaderboards of CSQA2.0 and StrategyQA. We also implement IA IAG-Student, the second variant that gets rid of dependencies on GPT service during inference, by training a student inductor model following two- step optimization scheme. Specifically, the model is firstly warmed up through distillation by using the GPT-3 statements as pseudo labels, and then fur- ther optimized with a novel TAILBACK approach: gradien'T bAck-propagation dIfferentiabLe BeAm sCores feedbacK. TAILBACK implements dif- ferentiable beam search algorithm for the inductor and allows the feedback from the generator to be back- pack-propagated to the inductor beam scores. We verify that IAG-Student improves over RAG baselines on a smaller architecture.
Related Work
Open-Domain Reasoning QA. Retrieval-based
approaches (Guu et al., 2020; Lewis et al., 2020) have been extensively explored cope with ODQA tasks (Chen al., 2017). Recent studics have focused solving ODQA tasks that require cer- tain sorts of reasoning, either explicitly implic- itly. Explicit reasoning tasks, such as Hotpot QA (bridge setting) (Yang et al., 2018), require the models to iteratively decompose the question and fetch new evidence until arriving at the answer. Im- plicit reasoning tasks, such StrategyQA (Geva et al., 2021) and CSQA (without commonsense knowledge base) (Talmor al., 2021), be more difficult solve since the answers are generally not present as plain text in web pages knowledge bases. To solve implicit reasoning tasks, two kinds of methods have been proposed, i.e., RAG and prompting. The former retrieves evidence based on the decomposed questions and relies on the gener- ator to implicitly reason over the evidence (Perez et al., 2020). But this approach becomes fragile when retrieved documents contain too little useful information too much noise. The latter teaches L.Mss to reason over questions by few-shot demon- stration (Wei et al., 2022; Chowdhery et al., 2022). But prompting templates are generally task-specific

---

and LLMs are prone to hallucinations. Our pro- posed framework combines the advantages of these two methods.
3.2 Knowledge Elicitation via Inductive Prompting
Recently, exploiting LLMs (c.g., GPT-3) for QA tasks has attracted increasing research interest due to their abilities to store factual knowledge and per- form reasoning. Representing knowledge as free- form text qualifies LLMs high-coverage knowl- edge bases for various domains, but the knowledge elicited from LLMs is prone to factual errors that can be detrimental to downstream tasks. Existing approaches to this problem either focus on arith- metic reasoning or commonsense reasoning tasks that involve multiple reasoning steps (Wei et al.. 2022) or cannot be easily adapted to open-domain settings (Creswell et al., 2022; Kazemi et al.. 2022),
Prompting-Basee -aaee Reasoning. Few-shot prompt- ing mainstream approach to eliciting knowl- edge from LLMs (Brown et al., 2020; Chowdhery et al.. 2022: Ma et al.. 2022; Liu et al., 2022a). To answer implicit reasoning questions, various prompting methods have been proposed among which CoT (Wei et al., 2022) has attracted the most research interest in the earlier research. It allows LLMs to answer complex questions in step-by- step manner. Follow-up studies include enhancing the consistency of CoT (Wang et al., 2022) and improving the correctness of the reasoning steps. Recently, adaptive prompting methods have been shown to achieve better performance on reason- ing tasks such bABI (Weston et al., 2016) and ProofWriter (Tafford et al., 2021). For example, the Selection-Inferrecce (SI) framework (Creswell et al., 2022) alternates between selection and infer- ence to generate a series of interpretable reasoning steps. LAMBADA (Kazemi et al., 2022) proposes the idea of backward chaining and employs four LLM-based sub-modules for reasoning. However, SI and LAMBDA rely on heuristic search algo- rithms performed in limited context space, which cannot be easily adapted to open-domain reasoning tasks. The inductive prompting proposed in this paper can be used for general-purpose knowledge elicitation.
To enhance the credibility of the statements gen- erated by LLMs, we propose a prompting method that is intuitively inspired by the idea of induc- tive reasoning. Inductive reasoning is a method of logical thinking that draws general conclusions from specific observations, during which analogy and generalization are two fundamental cognitive tools. Analogical reasoning allows us to establish connections between two similar objects and infer new information about one object based on what is known about the other. Generalization involves exploiting category information to generalize from the known to the unknown. As an example, con- sider the question shown in Figure 1. By analogical reasoning, one might conjecture that jellyfish, just like crabs and shrimps, are seldom found in dump- sters because these animals are similar in terms of inhabitants. By generalization, the fact that jelly- fish are aquatic animals can support the hypothesis that they live in the water instead of in dumpsters.
3 Induction-Augmented Generation
Based on the above cognitive functions, we pro- pose a prompting method that guides LLMs to gen- crate knowledge statements w.r.t. the question by building a two-step reasoning path, formally given as:
3.1 Overview
As illustrated in Figure 1, IAG enhances the con- ventional RAG architecture with an inductor that provides inductive knowledge for the generator to predict the answer. The inductor takes the question input and outputs knowledge statements in the form of inductive reasoning. These statements, to- gether with the retrieved documents, used as supporting evidence to feed into the generator.
Question: A statement about target.
Knowledge: Target, analog#1, analog#2 are hy- pernym. An assertion about hypernym.
The knowledge statement is formulated as a rea- soning path consisting of two sentences. The first sentence categorizes the target into its conceptual hypernym by associating two analogs that share some commonalities. And the second sentence states a fact about the hypernym that is contextually related to the topic of the question.
Section 3.2 introduces the inductive prompting method used for eahancing the factuality of the knowledge elicited from L.MSs. Two implementa- tions of our proposed IAG framework, i.c., IAG- GPT and IAG-Student are present in Section 3.3.

---

As example, consider the question shown in Figure 1. Regarding jellyfish as target, its analogs are crabs and shrimps, and its hypernym is aquatic animals. Hence, the reasoning path can be con- structed as follows: Jellyfish, crabs and shrimps are aquatic animals. You can't it catch aquatic ani- mals in the dumpster. We follow the above induc- tive reasoning pattern and construct prompting template consisting of 5 in-context demonstrations. The template is presented in Appendix A.
novel scheme TAILBACK that back-propagates the feedback from the generator the inductor via differentiable beam scores.
Step 1: Distillation For each question-answer
pair (g. a²) in the training set, we sample N differ- ent knowledge statements from GPT-3 using induc- tive prompting described in Section 3.2. The gener- ated statements for the question are denoted as KC {Kn {Ka) Besides, each question-ance -answer pair is accompanied by the top-M documents ranked by the retriever, represented as R = {Bm]m=
3.3 IAG Implementations
Instead of directly supervising the inductor using all the knowledge statements generated by GPT- 3, we claim that different statements should be distinguished according to their respective confi- dence during distillation. The confidence of each statement can be measured by the probability of predicting the ground-trult answer when used as supporting evidence for the generator.
3.3.1 IAG-GPT
For IAG-GPT, the function of induction is fully delegated to the GPT-3 service API. We leverage the prompting method described in Section 3.2 to generate inductive knowledge for each question. Inspired by (Wang ct al., 2022), we proposed to enhance the validity of the result by aggregating multiple knowledge statements. However, instead of explicitly voting on multiple results, we let the generator implicitly reason over the collection of all sampled knowledge statements.
Firstly, we adapt the generator to the task- specific dataset by fine-tuning for few steps using only the retrieved documents as supporting evi- dence. To calculate the confidence values of the N knowledge statements, fed each of them extra supporting evidence alongside M retrieved docu- ments into the generator, and compute the proba- bility of the ground-truth answer as
Specifically, for each question, we sample M inductive knowledge statements from GPT-3 and append them to the N retrieved documents, lead- ing to collection of M + N pieces of evidence. The generator, during both training and inference, takes all the M + N evidence input following the ission-in-decoder (FiD) approach (Izacard and Grave, 2021).
Pr = pp(a"(0gen q: Kn, R), n E [1, (1)
where yen is the parameters of the generator. We derive the confidence values [ix] by normal- izing the above probability distribution (pa) following
There are two benefits to feeding multiple sam- pled knowledge statements into the generator. Dur- ing training, the generator learns to predict the cor- rect answer based on diverse set of knowledge statements, making it robust to the noise present in the provided evidence. During inference, pro- viding richer set of evidence avoids the problem of local optimality in greedy decoding and has better chance of yielding the knowledge statement that best prompts the generator.
exp(1() Dn E exp(r())
Pn
P/2
(2)
where i and a are the mean value and standard deviation of the distribution Jo respectively. We found that the normalized values better reflect the difference among the knowledge statements.
Finally, we train the inductor using the knowl- edge statements from GPT-3, taking into account their confidence values. We propose two distilla- tion strategies that construct different distillation loss. The first strategy, termed QMax. allows only one knowledge statement for each question (i.e., the one with the maximum confidence) to con- tribute the loss, which is formulated as
3.3.2 IAG-Student
To get rid of the dependencies on GPT-3 during in- ference, we replace GPT-3 with student inductor model (we refer to it as inductor for brevity when there's no confusion). The inductor is trained in steps. In the first step, warm it up via distillation method, i.e., the knowledge statements elicited from GPT-3 are used as pseudo labels to train the inductor with a seq2seq loss. In the sec- ond step, we perform further optimization using
La =
Σ MIEKA
(0 mds q, Bet)),
(3)

---

Retrieval
Generation
Retriever
Rs. Ru
Generator
Induction
+g
Rai
FiD
Generator
question
R
Inductor
Beum Search
R
Generator
3t
20
= -log Σ Psti
P5 Likethood
plo.q, R...q.R..R.......
BP via DiIT Boam Scores
Illustration of our proposed TAILBACK training scheme. The likelihood values of the ground answer conditioned on different knowledge statements are used as feedhack from the generator, which can back-prop to the inductor via differentiable beam scores.
Figure 2:
where = argmax, Pn and Oind represents the parameters of the student inductor model.
Algorithm 1 TAILBACK Optimization
Input:
For the second strategy @weight- all the N state- ments of each question can contribute to the total loss, but are weighted by their confidence values, given by
D: training dataset
T: number of training iterations
1: for t <- to do
2:
(q, a+) s- leggee__ann -ge__sample())
3:
(Kn: } t beam_ beem__eercch(( (flndd
Weightt
(
Σ BrEK,
Dog(p(ye)oand 4, (ct)))
4:
{Rm J.M {Rm)==
get (od_docmmen((g)
for n + to do
Pn + -(p * 10 q, Kn BBm 1 mm)))
end for
Step 2: TAILBACK After warmed up through distillation, the inductor is further optimized us- ing novel TAILBACK training scheme that back- propagates the end-to-end loss to the inductor via differentiable beam scores, depicted in Figure 2. Intuitively, TAILBACK calculates the prediction loss of the generator based on the knowledge state- ments produced by the inductor and steers the model toward generating the statement that yields minimal prediction loss. A major challenge of this training scheme is that the generator is condi- tioned on discrete sequence of tokens produced by the student inductor model via outo-regressive decoding, which prevents the end-to-end loss from updating the parameters of the inductor through gradient descending.
(pn (Pn) t- nonoooioozeeeppe (1))
9: loss + -N 1
10:
(ussbackuard))
11: end for
Now we can describe the workflow or the TAIL- BACK optimization scheme. The pseudo-code can be found in Algorithm 1. First, given a question q. use differentiable beam search to generate N knowledge statements denoted IC = {Kn 1.V We normalize the beam scores of N knowledge statements using softmax, denoted
Sn = softmax (p( R 0inds 4)), n NJ. (5)
Then, we derive the end-to-end feedback from the generator by computing the probability of predict- ing the ground-truie answer given the question, M retrieved documents, and each generated knowl- edge statement. Finally, the loss of the student inductor model is constructed as
To address this issue, we implement a differ- entiable beam search algorithm for the inductor where the beam scores can be back-propagated. Specifically, the differentiable algorithm differs from the default implementation in that it pre- serves the computational graphs for deriving the beam scores instead of converting them into non- differentiable scalars during computing. Hence, the gradients are allowed to be propagated through the inductor via the beam scores.
ATAILBACK log Σ SG SG[p(a" 10gens
(6)
==1
9, kn, R)] ppKKn 9)-

---

where SG[.] is the stop gradient operator, i.e. the parameters of the generator won't t be updated dur- ing the optimization of the inductor.
5 Results
5.1 Main Results
After TAILBBAK optimization, different knowledge statements from the inductor and use them together with the retrieved documents to fine-tune the generator, just the same way as IAG-GPT.
sample N
We first report the performance of IAG-GPT comparison with SOTA models, as presented Table 1. It shows that the improvement of IAG over SOTA (74.1 - 78.2 for CSQA2.0 and 69.4 72.9 for StrategyQA is significant compared with the improvement of SOTA over previous SOTA (72.4 + 74.1 for CSQA2.0 and 65.4 69.4 for StrategyQA).
4 Experimental Setup
4.1 Datasets
For IAG-GPT, we report the scores of three dif- ferent setups of supporting evidence fed to the gen- erator. LAG-GPT outperforms existing methods on CSQA2.0 and Strategy QA. Besides, scores on the randomly held-out subsets of two tasks show that IAG-GPT has significant advantages over ChatGPT (version up Jan 29, 2023) in answer- ing reasoning questions. This result suggests that LLMs, even as strong as ChatGPT, can suffer from hallucination problems without access to retrieved factual evidence. Hence, combining prompting methods with information retrieval empowers IAG- GPT to answer reasoning questions more reliably. We show in Table 8 some cases where inductive knowledge helps predict the answer.
We evaluate IAG on two open-domain QA bench- marks, namely, CSQA2.0 (Talmor et al., 2021) and StrategyQA (Geva et al., 2021), both of which are binary classification tasks. CSQA2.0 consists of 14343 examples about everyday commonsense knowledge (9264/2541/2473 for train/dev/tst), and Strategy QA is multi-hop QA task comprised of 2780 examples (2290/490 for train/test) which requires implicit reasoning to solve. Note that since the official StrategyQA dataset doesn't include a development set, we randomly draw 1/4 of the ex- amples out of the training set for evaluation.
4.2 Models
Notably, our most powerful models have won the first place on the official leaderboards of CSQA2.0 and StrategyQA. Specifically, our IAG-GPT us- ing 10 retrieved documents and knowledge state- ments, enhanced with an assembling method, has set new record of 78.08* on CSQA2.0. For Strat- egyQA, the single model of IAG-GPT using 5 retrieved documents and knowledge statements achieves SOTA score of 72.86".
This section describes our setups for different com- ponents of IAG.
Retriever. For StrategyQA, the retriever is im- plemented sparse BM25 algorithm followed by a single-stream reranker (Gao et al., 2021). The dataset is accompanied by a corpus of context docu- ments as well as several annotated facts correspond- ing to each question, which are used to optimize the reranker model. For CSQA2.0, we input the questions into Google Search and use the top-5 snippets of the returned results as the retrieval data. Inductor. For IAG-GPT, the inductor is imple- mented by invoking the GPT-3 service API (text- davinci-003). We employ sampling method with a temperature of 0.7. For IAG-Studeni, we adopt T5-Large as the backbone of the student inductor model.
5.2 Prompting Methods
To verify the effectiveness of inductive prompt- ing, we compare IAG-GPT with two other baseline prompting methods. The first baseline is an ablated version of IAG-GPT that elicits knowledge from GPT-3 using a trivial prompt without inductive rea- soning. The template for trivial prompting is pre- sented in Table 6. The second baseline directly derives answers from GPT-3 via thain-of-though (CoT) prompting using the prompt proposed in their original paper (Wei al., 2022). For the sec- ond baseline, we experiment with two implemen-
Generator. For IAG-GPT, we initialize the gen- erator with T5-11B backbone. For IAG-Student, since fitting the inductor and the generator mod- ules pose high requirements on the GPU memory beyond our hardware capabilities, we resort a smaller architecture and use T5- L5-Large for the gen- erator as well.
'https://leaderbboad.. allenai. ort.m//aior////qa//// submissioossppblicc
'nttps://leaderrboard allenai. het.laaii..gg/ttateegga///////// submissionsspublic

---

Table Performance on two ODQA tasks. The first two columns report scores on CSQA2.0 dev set and Streategy QA test set respectively. The last two columns compare IAG with ChatGPT randomly held-out subset containing 50 examples for each task.
CSQA2.0 dev
StrategyQA test
CSQA2.0 dev*
StrategyQA dev
Method
DisentangledQA (Liu et al., 2022c)
69.4
UL2 (Tay et al., 2022)
59.0
Auto-CoT (Zhang al., 2022)
65.4
UNICORN-1 IB (Talmor al., 2021)
74.0
T5-11B (Talmor et al., 2021)
74.1
GKP (Liu al., 2022b)
72.4
ChatGPT (Ouyang et al., 2022)
60.0
52.0
Retrieval Only
77.2
70.0
IAG-GPT
Induction Only
74.0
71.2
(T5-11B Generator)
Retrieval Induction
78.2
72.9
80.0
74.0
tations including 1) the original implementation based on greedy decoding and 2) improved vari- ant (Wang al., 2022) that aggregates 5 reasoning traces via majority voting.
that it can fail and generate faulty statements some- times. Take the third question listed in Table 8 not
example, GPT-3 claims that "Beverages capable of drowning someone". As another exam- ple, given the question "Cotton candy is sometimes made out of cotton?". GPT-3 generates the follow- ing statements "Cotton, wool, and silk are fabrics. Cotton candy is made out of spun sugar or fabric". We attribute these failures to the fact that, although indutive prompting helps establish the connection between concept and its hypernym, correctly pre- dicting fact related the hypernym still depends on the internal knowledge of the language model, which is error-prone for tricky or long-tail ques- tions.
Table Comparison between IAG-GPT and CoT prompting. Scores are reported for the StrategyQA dev set.
Method
Accuracy
Greedy
71.5
CoT
Self-Consistency
73.3
IAG
w/o retrieval
73.6
(Trivial)
w/ retrieval
74.8
IAG
w/o retrieval
75.5
(Inductive)
w/ retrieval
76.2
5.3 Optimization of Inductor
For IAG-Student, the inductor model is optimized following the two-step training scheme as de- scribed in Section 3.3.2. This experiment provides insights into this training process.
The experiment is conducted on the StrategyQA dev set and the scores are reported in Table 2. As shown, our proposed IAG framework outperforms CoT methods by a large margin. This result indi- cates that compared to relying entirely on GPT-3 for answer prediction, combining the knowledge elicited from GPT-3 with information retrieval is better way of utilizing LLMs for QA tasks. Also, we find that inductive reasoning is better than trivial prompting providing the generator with useful knowledge for answer prediction. Table 9 lists some cases comparing different methods.
5.3.1 Distillation Strategies
In Section 3.3.2, we propose two strategies for the warmup training of the inductor, i.c., QMax and Qweight A prior study (Liu et al., 2022a) employs straightforward strategy that uses all these state- ments for training, denoted as QAI- The end-to-end performance of IAG-Student is evaluated by adopt- ing three different distillation strategies. We also introduce an RAG baseline that doesn't 't leverage inductive knowledge for prediction.
Limitations of Inductive Prompting. Although inductive prompting proves to be more effective in producing insightful knowledge, we have to admit
As shown in Figure 3, QAI hardly outperforms the RAG baseline (64.3 > 64.5). This can be at-

---

65.8
is observed on CSQA2.0. Since most questions in CSQA2.0 can be readily answered by using the retrieved documents, the inductive knowledge be- come less useful. However, solving StrategyQA re- quires implicit reasoning over multiple documents. When the retrieved documents fail to provide useful clues, our inductor can compensate for the missing information.
65.7
65.6
65.4
Accuracy
65.2
65.1
64.8
64.6
64.5
64.4
64.3
Nevertheless, observe consistent perfor- mance improvements brought by the further TAIL- BACK training on top of distillation on both CSQA2.0 (60.5 61.9) and StrategyQA (65.7 >> 66.6). This result proves that TAILBACK indeed steers the inductor towards producing knowledge statements that better prompt the generator. Quali- tative analysis is provided in Table 10.
64.2
induction
Qan
Oveur
Owarrt
Warmup Strategy
Figure 3: Comparison of different warmup strategies. Scores are reported on the Strategy QA dev set.
tributed to the fact that some knowledge statements sampled from GPT-3 are less useful for answer pre- diction. Using these statements indiscriminately can inevitably introduce noise and disturb the opti- mization of the inductor. As for Max leveraging feedback from the generator allows the inductor to learn from the best knowledge statement, But the size of the training data set is much smaller (only fraction of & generated statements are used). In comparison, Wweight achieves the best performance among all the strategies by supervising the induc- tor with a more diverse of GPT-3 statements while suppressing the effect of low-contribution ones. Hence @weight is adopted for the rest of the experiments.
5.4 Knowledge Fusion Mechanism
5.4.1 Knowledge Fusion v.s. Self-Consistency
IAG fuses multiple sampled knowledge statements into the generator for prediction. In contrast, the self-consistency approach (Wang al., 2022) pro- poses to explicitly vote on different reasoning paths sampled from LLM. We implement the idea self-consistency on IAG by allowing the genera- tor to predict multiple answers based on individual knowledge statements and reach a consensus by majority voting.
5.3.2 TAILBACK
Table 4: Comparison between knowledge fusion and self-con coff-consistency.
To provide insightful analysis of the TAILBACK optimization scheme, we evaluate the performance of IAG-Student three different stages: 1) with- out any induction knowledge, 2) introducing the student inductor model trained by distillation, and 3) further optimizing the model with TAILBACK Note that both the inductor and the generator adopt the Large architecture in this experiment.
CSQA2.0 dev
Strategy QA dev
Method
w/o induction
77.2
72.7
Self-Consistency Knowledge Fusion
77.6
74.8
78.2
76.2
Table Performance of IAG-Student at different stages the dev sets of CSQA2.0 and StrategyQA.
Table 4 compares different methods including the retrieval-only baseline. The knowledge fu- sion mechanism obviously superior to the self- consistency approach. We hypothesize that fusing all knowledge statements allows the generator to have holistic view of all evidence and make in- formed decisions. As for seff-consistency. although the voting mechanism can eliminate minority errors to some extent, it's less reliable due to easier prop- agation of random crrors in the sampled statements to the conclusions.
Training Step
CSQA2.0
Strategy
Retrieval Only
61.8
64.3
Distiillaion
60.5 (-1.3)
65.7 (+1.4)
TAILBBACK
61.9 (+0.1)
66.6 (+2.6)
can be shown Table that, introducing the inductor model significantly promotes the perfor- mance on StrategyQA, whereas little improvement

---

5.4.2 Number of Knowledge Statements
and the proposed distillation and TAILBACK op- timization scheme has only been verified on the Ta-Large architecture. We leave as our future work to experiment our methods with larger models and various backbones.
Our implementation of IAG samples 5 knowledge statements to feed into the generator. To justify this design choice, we evaluate the performance of IAG-Student with varying statement numbers. As shown in Figure 4, IAG-Student achieves the best performance with the statement number be- tween 5 and 7. The performance drops when the sampling number is either too large or too small. On the one side, small sampling number makes the model prone to random errors. On the other side, sampling too many statements inevitably in- troduce more noisy information that could mislead the prediction. Hence our choice of 5 knowledge statements makes a reasonable trade-off.
References
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language models few-shot learners. In Conference on Neural Information Processing Sys- tems, NeurIPS.
65.8
65.6
63.4
63.2
65.0
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes, 2017. Reading wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics, ACL, pages 1870-1879.
64.8
64.6
64.4
64.0
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi. David Dohan, Shivani Agrawal. Mark Omernick, An- drew M. Dai. Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier- Heleter, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CoRR, abs/2204.02311.
Number Knowledge Statements
Figure 4: Scores of IAG-Student on Strategy dev set with different numbers of knowledge statements.
6 Conclusion
To tackle the problem that serrivval-baseed methods cannot provide sufficient knowledge for the genera- tor to answer implicit reasoning questions, we pro- pose a novel IAG framework that augments RAG with inductive knowledge elicited from language models. We first design an inductive prompting method that enhances the factuality of knowledge elicited from GPT-3. We further propose sophis- ticated optimization scheme that trains a student inductor model via distillation and TAA.BACK. Our results suggest that IAG outperforms RAG in an- swering implicit reasoning questions.
Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Delection-inferrnnc:: Exploiting large language models for interpretable logical reasoning. CoRR. abs/2205.09711 099112.
Limitations
This work has several limitations. First, IAG has ev- ident advantages over RAG only for questions that cannot be readily answered by the retrieved docu- ments. Otherwise, the performance boosts brought by providing inductive knowledge are less signif- icant. Second, the effectiveness of IAG-Student
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer. and Andrew McCallum. 2019. Multi-step retriever- reader interaction for scalable open-domain question answering. In 7th International Conference on Learn- ing Representations, ICLR.

---

Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. Re- think training of BERT rerankers in sulti-stage re- trieval pipeline. In Advances in Information Retrieval -43rd European Conference on IR Research, ECIR, volume 12657 of Lecture Notes in Computer Science, pages 280-286.
Xinyin Ma, Xinchao Wang. Gongfan Fang, Yongliang Shen, and Weiming Lu. 2022. Prompting to distill: Boosting data-free knowledge distillation via rein- forced prompt. In Proceedings of the Thirty-Fire International Joint Conference on Artificial Intelli- gence, IJCAI, pages 4202-4302
Mor Geva, Daniel Khashabi, Elad Segal. Tushar Khot, Dan Roth, and Jonathan Berant, 2021. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Trans. Assoc. Comput. Linguistics, 9:346-361.
Long Ouyang, Jeff Wu, Xu Jiang. Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. CoRR, abs/2203.02155.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wet Wei Chang. 2020. Retrieval augmented language model pre-training. 37th International Conference on Machine Learning. ICML, volume 119. pages
Proceedings of the
Ethan Perez. Patrick S. H. Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsuper- vised question decomposition for question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP. pages 8864-8880.
Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open do- main question answering. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, EACI, pages 874-880.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin. Yuxiang Wu. and Alexander H. Miller. 2019. Language mod- els as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP, pages 2463-2473.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP, pages 6769- 6781.
Devendra Singh Sachan, Siva Reddy, William L. Hamil- ton, Chris Dyer, and Dani Yogatama. 2021. End-to- end training of multi-document reader and retriever for open-domain question answering. In Conference on Neural information Processing Systems, NeurlPS, pages 25968-25981.
Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, and Deepak Ramachandran. 2022. LAM- BADA: backward chaining for automated reasoning in natural language. CoRR, abs/2212. 13894.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis. Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. -augeete-d generation for knowledge- -nntnggveeeeeiivee NLP tasks. In Conference on Neural Information Processing Systems, NeurIPS.
Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. 2021. Proofwriter: Generating implications, proofs, and abductive statements over natural language. In Find- ings of the Association for Computational I. Linguistics: ACL/IJCNLP. volume ACL/UCNLP 2021, pages 3621-3634.
Jiacheng Liu, Skyler Hallinan, Ximing Lu, Pengfei He, Sean Welleck, Hannaneh Hajishirzi, and Yejin Choi. 2022a. Rainier: Reinforced knowledge introspec- tor for commonsense question answering. CoRR, abs/2210.03078.
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bha- gavatula, Yoav Goldberg. Yejin Choi, and Jonathan Berant. 2021. Commonsenseqa 2.0: Exposing the limits of AI through gamification. In Proceedings of the Neural Information Processing Systems Track Datasets and Benchmarks 1. NeurIPS Datasets and Benchmarks.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Pe- ter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022b. Generated knowledge prompting for commonsense reasoning. In Pruceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), ACL, pages 3159-3169.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unifying lan- guage learning paradigms. CoRR, abx/2205.06 1311
Qian Liu, Xiubo Geng, Yu Wang, Erik Cambria, and Daxin Jiang. 2022c. Disentangled retrieval and rea- soning for implicit question answering. IEFE Trans- actions on Neural Networks and Learning Systems.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, and Denny Zhou. 2022. Self- consistency improves chain of thought reasoning language models. CoRR, abs/2203. 11711.

---

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903.
Besides, the results on different setups of IAG- GPT suggest that, the relative contributions of the retrieval and the inductive knowledge can be differ- ent, depending on the tasks. As shown in Table I, for CSQA2.0, higher scores are reported for re- trieval only than for induction only, while the result is contrary for StrategyQA, These results can be attributed to the fact that questions in StrategyQA (e.g., Would a cattle farmer be useful drum maker?) require better reasoning ability to answer. To fully verify the effectiveness of the induc- tive knowledge, we compare the performance of IAG based on two different settings: 1) using 10 retrievals and 2) using retrievals and 5 knowledge statements. The results are shown in Table 7. For CSQA2.0, using more retrievals yields a slightly better result, while for StrategyQA, introducing knowledge statements significantly boosts the per- formance. This finding is consistent with our pre- vious conclusion that our method works better for reasoning intensive intensiv QA tasks. For CSQA2.0, many commonsense questions can be solved by referring to plain-texi records by retrieving methods. How- ever, for StrategyQA, the answers are unlikely be directly extracted from the retrieved documents but require reasoning over multiple documents to obtain. In such cases, introducing inductive knowl- edge can be very useful.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. 4th In- ternational Conference on Learning Representations, ICLR.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Ben- gio. William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answer- ing. In Proceedings of the 2018 Conference Em- pirical Methods in Natural Language Processing, pages 2369-2380.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. CoRR, abs/2210.033993.
Fenghin Zhu, Wenqiang Lei, Chao Wang. Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive vey on open-domain question answering. CoRR, abs/2101.00074,
A Prompting Template
Our experiments used two prompting templates including an inductive The template used for induc- tive prompting is presented in Table 5. It consists of 5 demonstrations constructed based on inductive reasoning, appended by the question of interest. We also present in Table 6 the trivial prompting template that is used in Section 5.2.
This is also the reason why the marginal perfor- mance gains are more significant for Strategy than for CSQA2.0 when inductive knowledge is offered besides the retrieved documents.
B.2 Comparison among Prompting Methods
B Additional Experimental Results
Table 9 compares the reasoning traces of different prompting methods. Although CoT is regarded a reliable prompting method by constructing inter- mediate reasoning steps, it's also prone to errors that could occur at any point of the reasoning path. For example, the sentence Thus, Christmas trees are not dissimilar to deciduous trees is a false de- ductive result based on the previous statements, leading to a false conclusion. As shown in Table 2, although enhancing CoT with ceff-consistency im- proves performance (71.5 -> 73.3), the intrinsic problem identified above still plagues this approach. Let alone the advantage that IAG can utilize re- trieved documents as extra information for predic- tion (76.2), the ablated version that employs only the GPT-3 knowledge (75.6) still has a lead over CoT.
B.1 Comparison between Information Retrieval and knowledge Induction
Table 8 lists some cases where the retrieved docu- ments fail to provide informative evidence whereas the inductive knowledge stands out. Typical fail- ures of retrieval-bane based approaches can be catego- rized into two occasions. Firstly, when the knowl- edge required for answering the question is too scarce to be found, the retriever could fetch docu- ments that hardly match the semantics of the ques- tion (c.g., Q3). Secondly, when the question is too trivial and the answer is unlikely to be officially documented, the retrievals could contain specific cases that contradict the commonsense conclusion (c.g., brittle failure of steel for QI and unequal leg length for Q2).

---

Table The inductive prompting template used in the experiments.
Question: is safe to keep wolves as pets.
Knowledge: Wolves, tigers and lions are wild animals. Wild animals are generally dangerous.
###
Question: Bacon is healthy diet food.
Knowledge: Bacon, chips and cakes junk food. Junk food is not healthy.
###
Question: Pens are more expensive than cars.
Knowledge: Pens, erasers and paper are stationery. Stationery is cheaper than cars.
###
Question: People make furniture out of oak.
Knowledge: Oak, pine and beech are Wood. Wood can be used to make furniture.
###
Question: Fridges are often used in the wild.
Knowledge: Fridges, ovens and TVs are appliances. Appliances used in houses.
###
Question: (User Question}
Knowledge:
Table 6: The trivial prompting template used in the experiments.
Table Performance comparison between different settings on CSQA2.0 and StrategyQA dev sets.
Question: It is safe to keep wolves as pets. Knowledge: Wolves dangerous.
Setting
CSQA2.0
StrategyQA
10 retrievals
78.4
73.8
###
Question: Bacon is healthy diet food.
5 retrievals
78.2
76.2
Knowledge: Bacon not healthy.
+ 5 knowledge
#Н#
Question: Pens are more expensive than cars.
still suffers from factual errors (e.g., #D3 for Q1), which is probably due the limitation imposed by small model size. In comparison, further opti- mization via TAILBACK deviates the inductor from the pre-defined reasoning pattern, but the knowl- edge (e.g., #TI for Q2) can better guide the genera- tor in predicting the right answer.
Knowledge: Pens are cheaper than cars. ###
Question: People make furniture out of oak.
Knowledge: Oak can be used to make furniture.
###
Question: Fridges are often used in the wild.
Knowledge: Fridges are used in houses.
###
Question: {User Question}
Knowledge:
B.3 Effect of Inductor Optimization
We list some cases in Table 10 that compares the knowledge statements generated by the student in- ductor model different optimization stage. As shown, after TAILBACK training, the inductor pro- duces knowledge statements that are more consis- tent. Besides, we find that, although distillation training enables the inductor to grasp the induc- tive reasoning pattern, the generated knowledge

---

Table 8: A demonstration of knowledge clicited from GPT-3 for cases in the CSQA2.0 dev set. Text colored in red indicates factual errors or contradictions to the ground-truth answer, and green indicates supporting evidence.
Q & A
Retrieved Documents
Inductive Knowledge
QI:A tube is never brittle if made of steel.
#1. Under what circumstances does the brittle failure of steel take place?
#1. Steel is a material. Materials can vary in their properties, in- cluding brittleness. #2. Steel, iron and concrete are metals. Metals are not brittle.
Answer: Yes
Prediction (w/o induction): No Prediction (w/ induction): Yes
#2. The material becomes brit- tle and, in extreme cases, mere contact with molten metal
Q2:A person has legs the same size as each other.
#1 Unequal leg length is where the legs are either differ- ent lengths or
#1. Legs, arms and fingers are appendages. All appendages person usually the same size.
Answer: Yes
Prediction (w/o induction): No
Prediction (w/ induction): Yes
#2. Having one leg longer than the other moderately normal. The condition is...
#2. People have two legs. Legs are the same size as each other.
Q3:Juice is capable of drowning
#1. A physical dissertation on drowning: in which submersion, commonly...
#1. Juice, water and soda are liq- uids. Liquids can drown peo- ple.
someone.
Answer: Yes
Prediction (w/o induction): No Prediction (w/ induction): Yes
#2. Because infants and young children can drown in only few inches of water, even water-filled containers ...
#2. Juice, water and milk are bev- erages. Beverages are not capa- ble of drowning someone.
Table Cases from StrategyQA to demonstrate the differences among the three prompting methods. Green indicates statements that support the answer, while red indicates otherwise.
&& A
CoT Prompting
Trivial Prompting
Inductive Prompting
Christmas trees are ever- green trees. Deciduous trees are tree that lose their leave. Thus, Christ- mas trees are dissimi- lar to deciduous trees. the answer is no.
Christmas trees
Christmas trees, fir trees
Q1: Are Christmas trees dissimilar to deciduous trees?
are conifer trees.
and pine trees conif- erous trees. Deciduous trees are not conifer- ous.
Deciduous trees
are trees that lose
Answer: Yes
their leaves each year.
Disgust is human emo- tion. There is no evidence that frogs feel emotions. Thus, frogs don't feel dis- gust.
Scientists do not currently know whether frogs feel emotions.
Frogs, lizards and snakes are animals. Ani- mals can feel emotions such as disgust.
Q2: Do frogs feel
disgust?
Answer: Yes
The metric system uses meters. meter about 3.3 feet. Thus, the foot is part of the metric system. So the answer is yes.
The foot, inch and yard are Imperial units. Im- perial units are not part of the metric sys- tem.
Q3: Is the foot part of the metric system?
The foot not part of the metric system.
Answer: No

---

Table 10: Comparison of inductive knowledge generated by the student inductor at different optimization stages. Text colored in red indicates factual errors contradictions to the ground -tunn answer, and green indicates supporting evidence.
Q& A
Inductive Knowledge after Distillation
Inductive Knowledge after TAILBACK
#D1. Ontology is the study of the human mind. It requires knowledge and under- standing. A scalpel is not required for ontology.
#T1. Ontology is the study of philosophy and religion. Philosophy and religion don't 't require scalpel.
Q1: Does
ontology
require a
#D2. Ontology is the study of the inner workings human beings. It requires a scalpel.
#T2. Ontology is a branch of knowledge that studies the meaning and structure of human thought and behavior. It does not require scalpel to be used.
scalpel?
Answer: No
#D3. Ontology, astronomy and philoso- phy are science disciplines. Science dis- ciplines require scalpel.
#T3. Ontology is branch of science that deals with knowledge and understanding. Knowledge is not usually obtained with scalpel.
#D1. A Great Depression is a depres- sion. Depressions do not affect everyone.
#T1. People often think of the Great De- pression an economic crisis, not mental illness.
Q2: Can person be diagnosed with Great Depression?
#D2. The Great Depression period of extreme economic uncertainty. People with extreme financial difficulties can be diagnosed with a Great Depression. #D3. The Great Depression was a time of great economic change in the 1930s. A person can be diagnosed with a Great Depression.
#T2. Great Depression period of economic downturn in the United States. Economic downturns often happen dur- ing the Great Depression.
Answer: No
#T3, Great Depression, post World War II Depression and postwar economic de- pression are all periods of economic downturn. Economic downturns have drastic effects on individuals and their families.


Reviews:
[{"review_id": "gJTMXmoGmD", "reviewer": "Reviewer_1Ny9", "summary": "", "questions": "1) Can you explain the tailback workflow using a pseudo code with detailed steps? Currently, it's a bit difficult to follow.\n2) Are there any plan to switch to later GPT versions from GPT-3? What would be the implications in terms of evaluation outcomes ?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the insightful comments. We have carefully read these comments and realized that our paper should have presented the ideas and methods in a more clear way to alleviate these concerns. We have revised the paper by incorporating the reviewer's suggestions. Here's our point-to-point response to the reviewer's comments.\n\n1. **Comment**: \"Although the Student Inductor model is shown to surpass the benchmark, the explanation and the underlying working principle was a bit hard to follow\"\n\n    **Response**: We appreciate the reviewer's observation and our paper may not have clearly described this module. In fact, the student inductor model can be regarded as a cheap substitution of GPT-3. It is implemented as a seq2seq model that takes in the question and outputs knowledge statements. The model is firstly trained by imitating GPT-3, i.e., we use the statements generated by GPT-3 to finetune the model. Next, we further optimize the model using TailBack. We will explain TailBack in more detail, which is also presented in the response to the thrid comment.\n\n2. **Comment**: \"For tailback differential beam searching is used, but it is hard to follow what are the steps pertaining to it.\" \n\n    **Response**: We apologize for not presenting our method as clearly as intended. We find that the beam scores reflect the probabilities of generating each token during decoding and thus can be used for constructing the loss function. Since Pytorch's default implementation of beam search is not intended for back-propagating any loss after the generation is done, the intermediate process for calculating the scores turns matrices into scalars. Differentiable beam search is in fact a modification to the Pytorch's implementation. We keep the intermediate variables in its matrix form so that the calculated beam scores are also pytorch matrices instead of scalars. As a result, the beam scores can be used for the loss function and be back-propagated through the induction model. In light of this feedback, we will add a further description of the method to the revision.\n\n3. **Comment**: \"Can you explain the tailback workflow using a pseudo code with detailed steps? Currently, it's a bit difficult to follow.\"\n\n    **Response**: Sure, the pseudo code is presented as follows:\n\n    \\=======================================================\n\n    Procedure TailBack\n\n    $\\mathcal{D}$: trainig dataset\n\n    $Rtv$: the retriever module\n\n    $\\theta_{gen}$: the generator\n\n    $\\theta_{ind}$: the induction model\n\n    for i in {1...num_iterations}:\n       \n&emsp;&emsp; &emsp; // get a question-answer pair from the training set\n\n&emsp;&emsp; &emsp; $ (q, a*) \\leftarrow getSample(\\mathcal{D}) $\n\n&emsp;&emsp; &emsp; // induction module generates N statements based on the question, also get the beam scores of each statement.\n\n&emsp;&emsp; &emsp; $(K_1, s_1), (K_2, s_2), ..., (K_N, s_N) \\leftarrow diffBeamSearch(\\theta_{ind}, q) $\n\n&emsp;&emsp; &emsp; // get M documents from the retriever\n\n&emsp;&emsp; &emsp; $\\{R_1, R_2, ..., R_M\\} \\leftarrow Rtv(q)$\n\n&emsp;&emsp; &emsp; // use generator to calculate the possibilities of generating the golden answer \n\n&emsp;&emsp; &emsp;for K_i in {K_1, K2, ..., K_N}:\n\n&emsp;&emsp; &emsp; &emsp;&emsp;$p_i \\leftarrow seq2seq\\_loss(\\theta_{gen}, q, K_i, {R_1, R_2, ..., R_M}, a*)$\n\n&emsp;&emsp; &emsp;end\n        \n&emsp;&emsp; &emsp;// normalize the possibilities into confidence values using equation(2)\n        \n&emsp;&emsp; &emsp;$\\{\\tilde{p_1}, \\tilde{p_2}, ..., \\tilde{p_N}\\} = normalize({p_1, p_2, ..., p_N})$\n        \n&emsp;&emsp; &emsp;// calculator the loss value for the induction model\n        \n&emsp;&emsp; &emsp;$loss = \\sum_{i=1}^N \\tilde{p_i} * s_i$\n        \n&emsp;&emsp; &emsp;$loss.backward()$\n    \n&emsp;end\n\n&emsp;\\=======================================================\n  \n&emsp;As you suggest, we will add the pseudo code into the revision if the space permits.\n\n\n4. **Comment**: \"Are there any plan to switch to later GPT versions from GPT-3? What would be the implications in terms of evaluation outcomes ?\"\n\n    **Response**: We fully acknowledge the point that experimenting with more LLMs will attach more valuable insights to our paper. In fact, after OpenAI made the API for GPT-3.5 publicly available, we conducted a few extra experiments on the development set. However, we didn't observe consistent performance improvement. The reason might be that GPT-3.5 is better at following human instructions, but its reasoning ability doesn't improve that much. As for GPT-4, we noticed that it has achieved first place in some nlp leaderboards. Although we currently don't have access to GPT-4, we expect the knowledge statements generated by GPT-4 to be much more informative and helpful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The description of the methodology and building up the framework was well explained.\nEvaluation conducted on two large Open-domain QA benchmark datasets.\n", "reasons_to_reject": "1) Although the Student Inductor model is shown to surpass the benchmark, the explanation and the underlying working principle was a bit hard to follow.\n2) For tailback differential beam searching is used, but it is hard to follow what are the steps pertaining to it.", "questions_to_authors": "1) Can you explain the tailback workflow using a pseudo code with detailed steps? Currently, it's a bit difficult to follow.\n2) Are there any plan to switch to later GPT versions from GPT-3? What would be the implications in terms of evaluation outcomes ?", "paper_topic_and_main_contributions": "This paper is based on Induction Augmented Generation Framework that uses LLMs for the implicit reasoning approach. The framework outperforms baselines for Retrieval Augmented Generation and ChatGPT on two Open domain tasks.\n\nOverall Contributions : \n1) Novel inductive prompting method which improves the factuality of knowledge elicited from LLMs.\n2)  A GPT implementation of the framework that improves over strong baseline models and ChatGPT.\n3) A TAILBACK optimization algorithm that trains the inductor which allows IAG-Student to outperform Retrieval Augmented Generation baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LIuV4fZqAO", "reviewer": "Reviewer_bY1q", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for the reviewer's comment highlighting the overall design of our framework and experiments. The valuable insights have prompted us to reconsider our method and to further improve the presentation. In this response, we address each of the reviewer's concerns and explain our viewpoint with additional supporting information.\n\n1. **Comment**: \"The proposed Induction-Augmented Generation (IAG) framework involves multiple components, including retrieval, induction, and generation, which might make it challenging for researchers to reproduce and implement the approach.\"\n\n    **Response**: We acknowledge that our implementation of the IAG framework seems a little complicated, which might be due to the training protocol. We have to clarify the IAG framework itself merely adds an induction module to the conventional RAG architecture for knowledge enhancement. When applying IAG to real-life applications, the induction module can be implemented using any off-the-shelf language models (GPT series, llama2, etc.). With the induction module frozen, the rest part of the IAG framework can be optimized exactly the same way as RAG. I hope this explanation can alleviate the concerns.\n\n2. **Comment**: \"The paper heavily relies on external language models, such as GPT-3, for generating inductive knowledge and improving performance. This reliance raises concerns about the availability, cost, and access to these models, which could limit the adoption of the proposed approach by researchers with limited resources or access.\"\n\n    **Response**: We sincerely appreciate the review's insightful comment on the use of GPT-3. We admit that dependencies on non-open-source services might jeopardize the integrity of academic research. Nonetheless, we noticed that, with LLMs (either open-source or non-open-source) becoming more powerful and more accessible to the public, a growing body of research has been focusing on the better utilization of these models for various research domains. Although we believe it is a trend to embrace, we do agree with the reviewer's opinion that academic research should focus on their own methods instead of relying on LLMs too heavily. As an effort to alleviate this dependency, we implemented IAG-Student in our work which makes the inference process self-contained.\n\n3. **Comment**: \"While the paper highlights successful cases where inductive knowledge enhances answer prediction, it does not thoroughly analyze or discuss cases where the approach might fail or provide incorrect answers. Understanding the limitations and potential pitfalls of the IAG framework is crucial for its safe and reliable application.\"\n\n    **Response**: We appreciate the reviewers' keen observation that our paper may not have analyzed the limitations in more detail. There are indeed cases where IAG can fail due to the introduction of false inductive knowledge, especially for some tricky questions. For example, GPT-3 can generate the following statements:\n\n    **Question**: Cotton candy is sometimes made out of cotton?\n\n    **Answer**: Cotton, wool, and silk are fabrics. Cotton candy is made out of spun sugar or fabric. \n    \n    This false induction might be caused by a superficial imitation of the reasoning pattern and the model's inability to capture the inner logic of such a tricky question. Although our previously submitted paper did include some cases (e.g., for Q3 in Table 6, GPT-3 generates logically conflicting statements) showing the weakness of our method, we overlooked the importance of an in-depth analysis. In light of this feedback, we'll incorporate more of such cases and analysis into the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper offers several strengths and benefits:\n\n1.The paper introduces a novel approach, Induction-Augmented Generation (IAG), which effectively combines inductive reasoning with language generation for answering implicit reasoning questions. \n\n2.  Implicit reasoning questions pose a significant challenge for open-domain question answering systems. By focusing on this challenge, the paper contributes to solving an important problem in the field of NLP, advancing the state of the art in understanding and generating reasoned answers.\n\n3. The paper presents a well-defined framework (IAG) and a detailed methodology for integrating inductive knowledge into the answer generation process.", "reasons_to_reject": "there are also some potential weaknesses:\n\n1.  The proposed Induction-Augmented Generation (IAG) framework involves multiple components, including retrieval, induction, and generation, which might make it challenging for researchers to reproduce and implement the approach. \n\n2.  The paper heavily relies on external language models, such as GPT-3, for generating inductive knowledge and improving performance. This reliance raises concerns about the availability, cost, and access to these models, which could limit the adoption of the proposed approach by researchers with limited resources or access.\n\n\n3.  While the paper highlights successful cases where inductive knowledge enhances answer prediction, it does not thoroughly analyze or discuss cases where the approach might fail or provide incorrect answers. Understanding the limitations and potential pitfalls of the IAG framework is crucial for its safe and reliable application.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper's key contributions are as follows:\n\n1. The paper proposes an inductive prompting method inspired by cognitive functions of inductive reasoning. This method guides language models (LLMs), specifically GPT-3, to generate knowledge statements that establish reasoning paths. \n\n2.  The paper introduces the Induction-Augmented Generation (IAG) framework, which enhances the traditional Retrieval-Augmented Generation (RAG) architecture. \n\n3. The paper presents two variants of the IAG framework. IAG-GPT leverages the inductive knowledge statements sampled from GPT-3 as evidence for the generator.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ttwwsOhAIT", "reviewer": "Reviewer_i5uE", "summary": "", "questions": "* Is the concatenation style of Retrieval Only in IAG the same as FiD [1]? If it is FiD-style, then there will be N number of passages to encode with the query. If not, as shown in concept figure 2, there will be a single long input which is a concatenation of the query and all N passages for IAG-GPT and IAG-student. It needs more clarification because in the knowledge fusion experiment, it seems like all of the question, knowledge statements, and retrieved documents are concatenated into a single long input.\n* On the CSQA dev dataset, it is stated that top-5 snippets are used (Line 423). However, in Line 465, on CSQA, it uses 10 retrieved documents. Is the top-5 snippets transformed into 10 retrieved documents? Or does it use top-10 snippets?\n* Inductive prompting generates implicit knowledge through in-context learning of 5 demonstrations. How are trivial and CoT prompting done? The examples are shown in Table 7, but it is not clear how the prompts are formed.\n* Are there cases where inductive knowledge statements are hallucinated?\n\n[1] Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We're grateful for the time and effort that you have taken to evaluate our work. Your expertise and attention to details have greatly contributed to the improvement of our paper. In this response, we would like to provide a detailed response to each of the raise points.\n\n\n1. **Comment**: \"The consistency in reporting the results needs to be done. Figure 3-4 and Table 2-4 uses StrategyQA dev which makes it hard to compare with Table 1 StrategyQA test baselines.\"\n\n    **Response**:  We sincerely apologize for any confusion or misunderstanding caused by our previous explanation/representation. Since both CSQA2.0 and StrategyQA provide blind test datasets, we have to upload our model predictions to the leaderboard website for evaluation, and re-submission is not allowed within 7 days to prevent overfitting. To conform to the policy, we decided to report our best results on the test dataset in Section 5.1 and perform the rest experiments on the development set for convenient local evaluation. But we emphasize that the scores reported for each experiment are consistent and can serve the experimental purposes.\n\n2. **Comment**: \"Table 2 shows that knowledge statements generated with inductive prompting support QA performance. However, to fully verify the effectiveness of inductive knowledge statements, additional comparisons need to be made on CSQA dev with 15 retrieved documents versus CSQA dev with 10 retrieved documents and 5 inductive knowledge statements. On StrategyQA, a comparison between 10 retrieved documents and 5 retrieved documents with 5 inductive knowledge statements needs to be conducted.\"\n\n    **Response**: Thanks for providing such valuable suggestions. We totally agree that it's important to compare IAG and the baseline with the same number of supporting evidence. As you point out, we conducted some extra experiments. For both datasets, we compare 10 retrieved documents and the combination of 5 documents + 5 knowledge statements. The results are as follows:\n\n    | Method | CSQA2.0 | StrategyQA |\n    |---|---|---|\n    | 10 retrievals | 78.4 | 73.8 |\n    | 5 retrievals + 5 knowledge | 78.2 | 76.2 | \n\n    As shown, for CSQA2.0, using 10 retrievals is slightly better, and for StrategyQA, using 5 retrievals and 5 knowledge statements is much better. This finding is consistent with our previous conclusion (lines 823-836) that our method works better for reasoning-intensive QA tasks. For CSQA2.0, many commonsense questions can be solved by accessing the powerful search engine such as Google. But for StrategyQA, the answers are unlikely to be directly found in documented records, and introducing inductive knowledge becomes very useful.\n\n3. **Comment**: \"Is the concatenation style of Retrieval Only in IAG the same as FiD [1]? If it is FiD-style, then there will be N number of passages to encode with the query. If not, as shown in concept figure 2, there will be a single long input which is a concatenation of the query and all N passages for IAG-GPT and IAG-student. It needs more clarification because in the knowledge fusion experiment, it seems like all of the question, knowledge statements, and retrieved documents are concatenated into a single long input.\"\n\n    **Response**: Yes, FiD is used for all experiments. We hypothesize that the confusion is caused by our presentation of figure 2 where we concatenate the query, knowledge statements and retrievals for simplicity. But actually, each piece of evidence is individually fed into the encoder following the FiD architecture. We'll improve the figure in the revision for clearer presentation.\n\n4. **Comment**: \"On the CSQA dev dataset, it is stated that top-5 snippets are used (Line 423). However, in Line 465, on CSQA, it uses 10 retrieved documents. Is the top-5 snippets transformed into 10 retrieved documents? Or does it use top-10 snippets?\"\n\n    **Response**: We sincerely regret our oversight in the clear description of the setups. In fact, using top-5 retrieved snippets is a consistent setup for all experiments, with only one exception as you point out in Line 465 where we use 10 snippets to achieve first place on the CSQA2.0 leaderboard. Since CSQA2.0 is an open-domain QA benchmark, there's no limitation to the use of external knowledge. So we enlarged the snippet number for the competition in order to achieve the best performance.\n\n5. **Comment**: \"Inductive prompting generates implicit knowledge through in-context learning of 5 demonstrations. How are trivial and CoT prompting done? The examples are shown in Table 7, but it is not clear how the prompts are formed.\"\n\n    **Response**: We apologize for not making the implementation details clear enough. For CoT prompting, the original CoT paper proposed a prompt used for the StrategyQA dataset, and we use exactly the same prompt. For trivial prompting, the few-shot demonstrations are constructed by directly presenting a conclusion without reasoning. For example:\n\n    **Question**: It is safe to keep wolves as pets.\n\n    **Knowledge**: Wolves are dangerous.\n\n    As you suggested, we will explain the CoT implementation more specifically and add the trivial prompting template to the appendix in the revision.\n\n\n6. **Comment**: \"Are there cases where inductive knowledge statements are hallucinated?\"\n\n    **Response**: You're right, there are such cases. Inductive reasoning can hallucinate for some strange or tricky queries. For example, GPT-3 can generate the following statement:\n\n    **Question**: Cotton candy is sometimes made out of cotton?\n    \n    **Knowledge**: Cotton, wool, and silk are fabrics. Cotton candy is made out of spun sugar or fabric.\n    \n    In such cases, the model can be easily misled by the query and hallucinate statements that only capture superficial reasoning patterns. We will take the advice and explicitly present these cases in the revision to provide the readers with more insights into the limitations of our method.\n\n7. **Comment**: \"Typos Grammar Style And Presentation Improvements\"\n\n    **Response**: Thanks for helping us find these problems. We'll fix them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed approach achieved state-of-the-art performance by integrating inductive knowledge into the prompt on the CSQA and Strategy QA datasets. The presentation of quality examples (such as Table 6 and 7) further supports the validity of the work.", "reasons_to_reject": "* The consistency in reporting the results needs to be done. Figure 3-4 and Table 2-4 uses StrategyQA dev which makes it hard to compare with Table 1 StrategyQA test baselines.  \n* Table 2 shows that knowledge statements generated with inductive prompting support QA performance. However, to fully verify the effectiveness of inductive knowledge statements, additional comparisons need to be made on CSQA dev with 15 retrieved documents versus CSQA dev with 10 retrieved documents and 5 inductive knowledge statements. On StrategyQA, a comparison between 10 retrieved documents and 5 retrieved documents with 5 inductive knowledge statements needs to be conducted.", "questions_to_authors": "* Is the concatenation style of Retrieval Only in IAG the same as FiD [1]? If it is FiD-style, then there will be N number of passages to encode with the query. If not, as shown in concept figure 2, there will be a single long input which is a concatenation of the query and all N passages for IAG-GPT and IAG-student. It needs more clarification because in the knowledge fusion experiment, it seems like all of the question, knowledge statements, and retrieved documents are concatenated into a single long input.\n* On the CSQA dev dataset, it is stated that top-5 snippets are used (Line 423). However, in Line 465, on CSQA, it uses 10 retrieved documents. Is the top-5 snippets transformed into 10 retrieved documents? Or does it use top-10 snippets?\n* Inductive prompting generates implicit knowledge through in-context learning of 5 demonstrations. How are trivial and CoT prompting done? The examples are shown in Table 7, but it is not clear how the prompts are formed.\n* Are there cases where inductive knowledge statements are hallucinated?\n\n[1] Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering", "paper_topic_and_main_contributions": "This paper introduces the Induction-Augmented Generation framework, which integrates inductive knowledge statements for Open-Domain QA tasks to enhance implicit reasoning. The framework proposes two models: IAG-GPT and IAG-student. IAG-GPT generates inductive knowledge statements using GPT-3 and utilizes both generated knowledge prompts and retrieved documents for QA tasks. Since IAG-GPT is dependent on the GPT-3 API, IAG-student is trained to distill GPT-3.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* In Line 467, naming of the dataset should be consistent\n\t* sudden acronym for StrategyQA -> SQA\n* Including the model size could strengthen your Table 1."}, "extra_text": {}}, {"review_id": "LKMFvBsLIp", "reviewer": "Reviewer_HrRJ", "summary": "", "questions": "1. In the IAG-student algorithm, the generator is first trained followed by the inductor, will finetuning the generator together with the inductor help?\n\n2. In line 228, How to calculate the mu and sigma for the distribution?\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the invaluable insights and suggestions you have offered on our paper. We have carefully reviewed these comments and tried our best to address your concerns. The point-to-point response is listed as follows.\n\n1. **Comment**: \"The generalization ability of the model is a major concern. The model assigns inductive information to every question, even when some questions do not require it. A potential solution could be implementing a question classifier to identify the type of question and determine whether inductive information is necessary for a particular query.\"\n\n    **Response**: Thank you for your insightful comment on the generalization of the method. You are right that not all questions require inductive information to answer. However, from our point of view, whether to use the inductive information depends not only on the question itself but also on the quality of the inductive knowledge. If we want to build a classifier to accurately determine if a question can be answered by the generator without introducing extra knowledge, does it mean that the classifier should have the same amount of knowledge as the generator? We think it might be more feasible to insert a validation module AFTER the induction module for validating the quality of the retrieved information and the inductive knowledge (e.g., detecting logically inconsistent or false statements and removing them), which we leave for our future work. However, we do agree that, from the engineering perspective, pre-detection of the necessity of induction can reduce system costs. This can be an important consideration for engineering implementations.\n\n\n2. **Comment**: \"The strict structure/formulation of the prompt, especially the Knowledge part, is another issue (lines 245-247).\"\n\n    **Response**: We appreciate your comment regarding the formulation of the prompt. We admit that our formulation of the inductive prompting doesn't seem too strict. Actually, given that the input questions are free-form text, it can be hard to present the reasoning process into a well-structured formulation. Just like Chain-of-thought prompting, the reasoning path can be constructed very differently for different kinds of questions. At the core of inductive prompting is the logical reasoning that projects the target into its conceptual hypernym by analogy, which has been explicitly demonstrated in the knowledge part (i.e., \"Target, analog#1, analog#2 are hypernym\"). Besides, we present a few examples of inductive prompting in Table 5, which will hopefully help the readers better understand the prompting method. Nonetheless, we agree that the formalization of the research problem can benefit from a more strict formulation of the reasoning path, and we'll keep exploring a better representation of the prompting structure, both for our inductive prompting and for other methods such as CoT.\n\n3. **Comment**: \"Another minor issue is that there is a huge gap between the performance of IAG-GPT and IAG-Student, which makes the distilled model and its corresponding algorithm less convincing. More experiments on larger models are expected.\"\n\n    **Response**: We fully acknowledge the point that larger induction models can narrow the performance gap. It's natural to expect a huge performance gap given that our student induction model (770m) is several orders of magnitude smaller than GPT-3 (170B). But our intention of introducing IAG-Student is not to make it comparable to the IAG-GPT, but to demonstrate that using a small add-on module can improve upon the RAG baseline. Besides, we have to admit that our current GPU infrastructure doesn't support training a 13B generator and a larger induction model simultaneously. As said in the limitations section, experimenting with larger models will be the direction of our future exploration.\n\n4. **Comment**: \"In the IAG-student algorithm, the generator is first trained followed by the inductor, will finetuning the generator together with the inductor help?\"\n\n    **Response**: Thanks for your insightful suggestion. In fact, we did conduct experiments that train both modules together, but it converges slower and the performance doesn't improve. In the first training stage, the generator is used to provide feedback signals for the student induction model to know which knowledge statement is better. If the generator is not finetuned in advance, it can give false feedbacks. We suppose this can be the reason for slower convergence and we decide to fine-tune the generator as the first step. I hope this explanation can address your concerns.\n\n5. **Comment**: \"In line 228, How to calculate the mu and sigma for the distribution?\"\n\n    **Response**: We would like to apologize for not adequately describing the details. The distribution is a vector of scalar where each scalar represents the probability of generating the golden answer given the corresponding knowledge statement. $\\mu$ and $\\sigma$ are the average and deviation of this distribution respectively and can be easily calculated in the usual way: $\\mu = \\frac{\\sum_{i=1}^{n}}{n}$ and $\\sigma=\\sqrt{\\frac{\\sum_{i=1}^n(x_i-\\mu)^2}{n}}$. In light of this feedback, we'll revise this paragraph to provide a clearer explanation of the method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper addresses a non-trivial problem for implicit reasoning by introducing an inductor with the assistance of LLMs.  \n\nThe authors conduct extensive experiments and the experiment results are reasonable.\n\n", "reasons_to_reject": "1. The generalization ability of the model is a major concern. \n- The model assigns inductive information to every question, even when some questions do not require it. A potential solution could be implementing a question classifier to identify the type of question and determine whether inductive information is necessary for a particular query.\n- The strict structure/formulation of the prompt, especially the Knowledge part, is another issue (lines 245-247). \n\n2. Another minor issue is that there is a huge gap between the performance of IAG-GPT and IAG-Student, which makes the distilled model and its corresponding algorithm less convincing. More experiments on larger models are expected. \n\n", "questions_to_authors": "1. In the IAG-student algorithm, the generator is first trained followed by the inductor, will finetuning the generator together with the inductor help?\n\n2. In line 228, How to calculate the mu and sigma for the distribution?\n\n", "paper_topic_and_main_contributions": "The authors propose an induction-augmented framework that utilizes inductive knowledge derived from LLMs and the retrieved documents for better implicit reasoning. Specifically, they enhance the conventional RAG with an inductor that generates inductive knowledge.\n\nThe authors propose an IAG-GPT model which directly utilizes GPT-3 and IAG-Student which is first trained via knowledge distillation with GPT-3 pseudo labels, and then optimized through a differentiable beam search algorithm.\n\nThe experiments show that IAG-GPT has significant advantages over ChatGPT and performs extremely well on CSQA2.0 and StrategyQA. IAG-Student outperforms RAG baselines.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes the Induction-Augmented Generation (IAG) framework, designed to enhance implicit reasoning in Open-Domain Question-Answering (QA) tasks. The framework includes two models: IAG-GPT and IAG-Student. IAG-GPT generates inductive knowledge statements using GPT-3 and combines these statements with retrieved documents for QA tasks. Removing the dependency on GPT-3 at inference time, the IAG-Student model is trained through knowledge distillation with GPT-3 pseudo labels and optimized using a differentiable beam search algorithm.

This paper presents a well-explained methodology and framework for solving the challenging problem of implicit reasoning in open-domain question answering. This work proposes a novel optimization scheme to train the inductor model through distillation and back-propagation of the generator feedback via differentiable beam scores. The authors conduct a convincing evaluation on two open-domain QA benchmark datasets (CSQA2 and Strategy QA) and achieve state-of-the-art performance on both (at least as of January 2023 for Strategy QA).

As highlighted by the reviewers, the authors could further strengthen their contribution by conducting, for example, a more thorough analysis of the failure modes of the approach, of the generalization of the method, and of the effectiveness of the inductive knowledge statements.
Decision: Accept-Main