Introduction
Open-Domain Question Answering (ODQA) Zhu et al. ( 2021 ) has attracted increasing research attention.
Compared with the closed-domain setting, techniques developed upon ODQA models can empower search engines with the ability to respond to a wider range of user queries.
As a typical knowledge-intensive task, ODQA has been extensively studied within the scope of information retrieval Karpukhin et al. ( 2020 ); Das et al. ( 2019 ); Sachan et al. ( 2021 ) , where access to external knowledge sources such as web pages or knowledge bases is required.
Another line of research exploits large language models (LLMs) such as GPT-3 and PaLM as the knowledge source Petroni et al. ( 2019 ); Liu et al. ( 2022b ) and develops various prompting methods to elicit knowledge from LLMs that is implicitly encoded in the parameters.
However, to answer reasoning questions, i.e., questions that demand some degree of reasoning ability, either retrieval-based or prompting-based approaches suffer from their intrinsic limitations.

On the one hand, although RAG Lewis et al. ( 2020 ) has become the SOTA architecture for ODQA, documents retrieved from common knowledge bases generally suffer from the limitations of constrained coverage and noisy information,
especially for implicit reasoning questions whose answers are not well documented.
For example, it’s trivial for an average child to answer questions such as "can you catch a jellyfish in the dumpster?" . However, it’s unlikely to find the answer directly from the web or books.
As shown in Figure 1 , the retrieved documents can hardly answer the question in such cases. Hence, relying entirely on information retrieval is insufficient to solve implicit reasoning questions.

On the other hand, prompting-based methods can exploit the considerable amount of knowledge encoded in the parameters of LLMs for QA tasks Wang et al. ( 2022 ); Wei et al. ( 2022 ); Chowdhery et al. ( 2022 ); Brown et al. ( 2020 ) .
But the problem of hallucination (i.e., generating natural language statements that are factually incorrect) imposes limitations on LLMs in terms of factuality and credibility.
To better control the knowledge elicited from LLMs, various prompting methods such as chain-of-thought (CoT) Wei et al. ( 2022 ) have been proposed by constructing intermediate reasoning steps until arriving at the conclusion.
However, capability of the an LLM is intrinsically constrained by its parameter size, making it unable to respond correctly to domain-specific questions beyond the scope of its training corpus.

In view of the above challenges, it requires a new paradigm for building models applicable to reasoning QA.
To this end, we combine the advantages of these two kinds of methods and propose IAG , an I nduction- A ugmented G eneration framework for answering implicit reasoning questions.
IAG enhances the conventional RAG with an inductor that generates inductive knowledge w.r.t each question.
To derive such knowledge, we propose a novel prompting method, which is intuitively inspired by the cognitive functions of inductive reasoning, to elicit inductive knowledge from an LLM (i.e., GPT-3).
Our first implementation of IAG, which is termed IAG-GPT, directly leverages the knowledge statements sampled from GPT-3 as evidence alongside the retrieved documents to feed into the answer generator. We show that IAG-GPT improves over SOTA models on multiple ODQA datasets and has won the first place in the official leaderboards of CSQA2.0 and StrategyQA.
We also implement IAG- Student , the second variant that gets rid of dependencies on GPT service during inference, by training a student inductor model following a two-step optimization scheme. Specifically, the model is firstly warmed up through distillation by using the GPT-3 statements as pseudo labels, and then further optimized with a novel TailBack approach: gradien T b A ck-propagation d I fferentiab L e B e A m s C ores feedbac K . TailBack implements a differentiable beam search algorithm for the inductor and allows the feedback from the generator to be back-propagated to the inductor via beam scores.
We verify that IAG- Student improves over RAG baselines on a smaller architecture.

The contributions of this paper include: 1) an inductive prompting method which improves the factuality of knowledge elicited from LLMs by constructing a reasoning path via inductive reasoning; 2) an IAG-GPT implementation that improves over strong baseline models and ChatGPT by leveraging the knowledge elicited from GPT-3 as auxiliary supporting evidence for the generator; 3) a TailBack optimization algorithm to train the inductor which allows IAG- Student to outperform RAG baselines under a small model size .