RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Related Work
Open-Domain Reasoning QA. Retrieval-based approaches Guu et al. ( 2020 ); Lewis et al. ( 2020 ) have been extensively explored to cope with ODQA tasks Chen et al. ( 2017 ) .
Recent studies have focused on solving ODQA tasks that require certain sorts of reasoning, either explicitly or implicitly.
Explicit reasoning tasks, such as Hotpot QA (bridge setting) Yang et al. ( 2018 ) , require the models to iteratively decompose the question and fetch new evidence until arriving at the answer.
Implicit reasoning tasks, such as StrategyQA Geva et al. ( 2021 ) and CSQA (without commonsense knowledge base) Talmor et al. ( 2021 ) , can be more difficult to solve since the answers are generally not present as plain text in web pages or knowledge bases.
To solve implicit reasoning tasks, two kinds of methods have been proposed, i.e., RAG and prompting.
The former retrieves evidence based on the decomposed questions and relies on the generator to implicitly reason over the evidence Perez et al. ( 2020 ) . But this approach becomes fragile when retrieved documents contain too little useful information or too much noise.
The latter teaches LLMs to reason over questions by few-shot demonstration Wei et al. ( 2022 ); Chowdhery et al. ( 2022 ) . But prompting templates are generally task-specific and LLMs are prone to hallucinations.
Our proposed framework combines the advantages of these two methods.

Prompting-Based Reasoning. Few-shot prompting is a mainstream approach to eliciting knowledge from LLMs Brown et al. ( 2020 ); Chowdhery et al. ( 2022 ); Ma et al. ( 2022 ); Liu et al. ( 2022a ) .
To answer implicit reasoning questions, various prompting methods have been proposed among which CoT Wei et al. ( 2022 ) has attracted the most research interest in the earlier research. It allows LLMs to answer complex questions in a step-by-step manner.
Follow-up studies include enhancing the consistency of CoT Wang et al. ( 2022 ) and improving the correctness of the reasoning steps.
Recently, adaptive prompting methods have been shown to achieve better performance on reasoning tasks such as bABI Weston et al. ( 2016 ) and ProofWriter Tafjord et al. ( 2021 ) . For example, the Selection-Inference (SI) framework Creswell et al. ( 2022 ) alternates between selection and inference to generate a series of interpretable reasoning steps.
LAMBADA Kazemi et al. ( 2022 ) proposes the idea of backward chaining and employs four LLM-based sub-modules for reasoning.
However, SI and LAMBDA rely on heuristic search algorithms performed in a limited context space, which cannot be easily adapted to open-domain reasoning tasks.
The inductive prompting proposed in this paper can be used for general-purpose knowledge elicitation.
Overview
As illustrated in Figure 1 , IAG enhances the conventional RAG architecture with an inductor that provides inductive knowledge for the generator to predict the answer.
The inductor takes the question as input and outputs knowledge statements in the form of inductive reasoning.
These statements, together with the retrieved documents, are used as supporting evidence to feed into the generator.

Section 3.2 introduces the inductive prompting method used for enhancing the factuality of the knowledge elicited from LLMs. Two implementations of our proposed IAG framework, i.e., IAG-GPT and IAG- Student are present in Section 3.3 .
Knowledge Elicitation via Inductive Prompting
Recently, exploiting LLMs (e.g., GPT-3) for QA tasks has attracted increasing research interest due to their abilities to store factual knowledge and perform reasoning.
Representing knowledge as free-form text qualifies LLMs as high-coverage knowledge bases for various domains, but the knowledge elicited from LLMs is prone to factual errors that can be detrimental to downstream tasks.
Existing approaches to this problem either focus on arithmetic reasoning or commonsense reasoning tasks that involve multiple reasoning steps Wei et al. ( 2022 ) or cannot be easily adapted to open-domain settings Creswell et al. ( 2022 ); Kazemi et al. ( 2022 ) .

To enhance the credibility of the statements generated by LLMs, we propose a prompting method that is intuitively inspired by the idea of inductive reasoning.
Inductive reasoning is a method of logical thinking that draws general conclusions from specific observations, during which analogy and generalization are two fundamental cognitive tools.
Analogical reasoning allows us to establish connections between two similar objects and infer new information about one object based on what is known about the other.
Generalization involves exploiting category information to generalize from the known to the unknown.
As an example, consider the question shown in Figure 1 .
By analogical reasoning, one might conjecture that jellyfish, just like crabs and shrimps, are seldom found in dumpsters because these animals are similar in terms of inhabitants. By generalization, the fact that jellyfish are aquatic animals can support the hypothesis that they live in the water instead of in dumpsters.

Based on the above cognitive functions, we propose a prompting method that guides LLMs to generate knowledge statements w.r.t. the question by building a two-step reasoning path, formally given as:

Question: A statement about target .

Knowledge: Target , analog#1 , analog#2 are hypernym . An assertion about hypernym .

The knowledge statement is formulated as a reasoning path consisting of two sentences. The first sentence categorizes the target into its conceptual hypernym by associating two analogs that share some commonalities. And the second sentence states a fact about the hypernym that is contextually related to the topic of the question.

As an example, consider the question shown in Figure 1 . Regarding jellyfish as target , its analogs are crabs and shrimps, and its hypernym is aquatic animals.
Hence, the reasoning path can be constructed as follows: Jellyfish, crabs and shrimps are aquatic animals. You can’t catch aquatic animals in the dumpster. We follow the above inductive reasoning pattern and construct a prompting template consisting of 5 in-context demonstrations.
The template is presented in Appendix A .
IAG-GPT
For IAG-GPT, the function of induction is fully delegated to the GPT-3 service API.
We leverage the prompting method described in Section 3.2 to generate inductive knowledge for each question.
Inspired by Wang et al. ( 2022 ) , we proposed to enhance the validity of the result by aggregating multiple knowledge statements.
However, instead of explicitly voting on multiple results, we let the generator implicitly reason over the collection of all sampled knowledge statements.

Specifically, for each question, we sample M inductive knowledge statements from GPT-3 and append them to the N retrieved documents, leading to a collection of M+N pieces of evidence.
The generator, during both training and inference, takes all the M+N evidence as input following the fusion-in-decoder (FiD) approach Izacard and Grave ( 2021 ) .

There are two benefits to feeding multiple sampled knowledge statements into the generator.
During training, the generator learns to predict the correct answer based on a diverse set of knowledge statements, making it robust to the noise present in the provided evidence.
During inference, providing a richer set of evidence avoids the problem of local optimality in greedy decoding and has a better chance of yielding the knowledge statement that best prompts the generator.
IAG- Student
To get rid of the dependencies on GPT-3 during inference, we replace GPT-3 with a student inductor model (we refer to it as inductor for brevity when there’s no confusion).
The inductor is trained in two steps.
In the first step, we warm it up via a distillation method, i.e., the knowledge statements elicited from GPT-3 are used as pseudo labels to train the inductor with a seq2seq loss.
In the second step, we perform further optimization using a novel scheme TailBack that back-propagates the feedback from the generator to the inductor via differentiable beam scores.
Datasets
We evaluate IAG on two open-domain QA benchmarks, namely, CSQA2.0 Talmor et al. ( 2021 ) and StrategyQA Geva et al. ( 2021 ) , both of which are binary classification tasks. CSQA2.0 consists of 14343 examples about everyday commonsense knowledge (9264/2541/2473 for train/dev/test), and StrategyQA is a multi-hop QA task comprised of 2780 examples (2290/490 for train/test) which requires implicit reasoning to solve.
Note that since the official StrategyQA dataset doesn’t include a development set, we randomly draw 1/4 of the examples out of the training set for evaluation.
Models
This section describes our setups for different components of IAG.

Retriever. For StrategyQA, the retriever is implemented as a sparse BM25 algorithm followed by a single-stream reranker Gao et al. ( 2021 ) . The dataset is accompanied by a corpus of context documents as well as several annotated facts corresponding to each question, which are used to optimize the reranker model.
For CSQA2.0, we input the questions into Google Search and use the top-5 snippets of the returned results as the retrieval data.

Inductor. For IAG-GPT, the inductor is implemented by invoking the GPT-3 service API (text-davinci-003). We employ a sampling method with a temperature of 0.7.
For IAG- Student , we adopt T5-Large as the backbone of the student inductor model.

Generator. For IAG-GPT, we initialize the generator with a T5-11B backbone.
For IAG- Student , since fitting the inductor and the generator modules pose high requirements on the GPU memory beyond our hardware capabilities, we resort to a smaller architecture and use T5-Large for the generator as well.
Main Results
We first report the performance of IAG-GPT in comparison with SOTA models, as presented in Table 1 .
It shows that the improvement of IAG over SOTA (74.1 \rightarrow 78.2 for CSQA2.0 and 69.4 \rightarrow 72.9 for StrategyQA) is significant compared with the improvement of SOTA over previous SOTA (72.4 \rightarrow 74.1 for CSQA2.0 and 65.4 \rightarrow 69.4 for StrategyQA).

For IAG-GPT, we report the scores of three different setups of supporting evidence fed to the generator.
IAG-GPT outperforms existing methods on CSQA2.0 and StrategyQA.
Besides, scores on the randomly held-out subsets of two tasks show that IAG-GPT has significant advantages over ChatGPT (version up to Jan 29, 2023) in answering reasoning questions.
This result suggests that LLMs, even as strong as ChatGPT, can suffer from hallucination problems without access to retrieved factual evidence. Hence, combining prompting methods with information retrieval empowers IAG-GPT to answer reasoning questions more reliably. We show in Table 8 some cases where inductive knowledge helps predict the answer.

Notably, our most powerful models have won the first place on the official leaderboards of CSQA2.0 and StrategyQA.
Specifically, our IAG-GPT using 10 retrieved documents and 5 knowledge statements, enhanced with an assembling method, has set a new record of 78.08 * * * https://leaderboard.allenai.org/csqa2/submissions/public on CSQA2.0.
For StrategyQA, the single model of IAG-GPT using 5 retrieved documents and 5 knowledge statements achieves a SOTA score of 72.86 † † † https://leaderboard.allenai.org/strategyqa/submissions/public .
Prompting Methods
To verify the effectiveness of inductive prompting, we compare IAG-GPT with two other baseline prompting methods.
The first baseline is an ablated version of IAG-GPT that elicits knowledge from GPT-3 using a trivial prompt without inductive reasoning. The template for trivial prompting is presented in Table 6 .
The second baseline directly derives answers from GPT-3 via chain-of-thought (CoT) prompting using the prompt proposed in their original paper Wei et al. ( 2022 ) .
For the second baseline, we experiment with two implementations including 1) the original implementation based on greedy decoding and 2) an improved variant Wang et al. ( 2022 ) that aggregates 5 reasoning traces via majority voting.

The experiment is conducted on the StrategyQA dev set and the scores are reported in Table 2 .
As shown, our proposed IAG framework outperforms CoT methods by a large margin.
This result indicates that compared to relying entirely on GPT-3 for answer prediction, combining the knowledge elicited from GPT-3 with information retrieval is a better way of utilizing LLMs for QA tasks.
Also, we find that inductive reasoning is better than trivial prompting at providing the generator with useful knowledge for answer prediction.
Table 9 lists some cases comparing different methods.
Optimization of Inductor
For IAG- Student , the inductor model is optimized following the two-step training scheme as described in Section 3.3.2 . This experiment provides insights into this training process.
Distillation Strategies
In Section 3.3.2 , we propose two strategies for the warmup training of the inductor, i.e., \mathcal{Q}_{\text{Max}} and \mathcal{Q}_{\text{Weight}} .
A prior study Liu et al. ( 2022a ) employs a straightforward strategy that uses all these statements for training, denoted as \mathcal{Q}_{\text{All}} .
The end-to-end performance of IAG- Student is evaluated by adopting three different distillation strategies.
We also introduce an RAG baseline that doesn’t leverage inductive knowledge for prediction.

As shown in Figure 3 , \mathcal{Q}_{\text{All}} hardly outperforms the RAG baseline ( 64.3\rightarrow 64.5 ).
This can be attributed to the fact that some knowledge statements sampled from GPT-3 are less useful for answer prediction.
Using these statements indiscriminately can inevitably introduce noise and disturb the optimization of the inductor.
As for \mathcal{Q}_{\text{Max}} , leveraging feedback from the generator allows the inductor to learn from the best knowledge statement.
But the size of the training data set is much smaller (only a fraction of \frac{1}{N} generated statements are used).
In comparison, \mathcal{Q}_{\text{Weight}} achieves the best performance among all the strategies by supervising the inductor with a more diverse set of GPT-3 statements while suppressing the effect of low-contribution ones. Hence \mathcal{Q}_{\text{Weight}} is adopted for the rest of the experiments.
TailBack
To provide insightful analysis of the TailBack optimization scheme, we evaluate the performance of IAG- Student at three different stages: 1) without any induction knowledge, 2) introducing the student inductor model trained by distillation, and 3) further optimizing the model with TailBack .
Note that both the inductor and the generator adopt the T5-Large architecture in this experiment.

It can be shown in Table 3 that, introducing the inductor model significantly promotes the performance on StrategyQA, whereas little improvement is observed on CSQA2.0.
Since most questions in CSQA2.0 can be readily answered by using the retrieved documents, the inductive knowledge become less useful.
However, solving StrategyQA requires implicit reasoning over multiple documents. When the retrieved documents fail to provide useful clues, our inductor can compensate for the missing information.

Nevertheless, we observe consistent performance improvements brought by the further TailBack training on top of distillation on both CSQA2.0 (60.5 \rightarrow 61.9) and StrategyQA (65.7 \rightarrow 66.6).
This result proves that TailBack indeed steers the inductor towards producing knowledge statements that better prompt the generator. Qualitative analysis is provided in Table 10 .
Knowledge Fusion v.s. Self-Consistency
IAG fuses multiple sampled knowledge statements into the generator for prediction. In contrast, the self-consistency approach Wang et al. ( 2022 ) proposes to explicitly vote on different reasoning paths sampled from LLM. We implement the idea of self-consistency on IAG by allowing the generator to predict multiple answers based on individual knowledge statements and reach a consensus by majority voting.

Table 4 compares different methods including the retrieval-only baseline.
The knowledge fusion mechanism is obviously superior to the self-consistency approach.
We hypothesize that fusing all knowledge statements allows the generator to have a holistic view of all evidence and make informed decisions.
As for self-consistency, although the voting mechanism can eliminate minority errors to some extent, it’s less reliable due to easier propagation of random errors in the sampled statements to the conclusions.
Number of Knowledge Statements
Our implementation of IAG samples 5 knowledge statements to feed into the generator.
To justify this design choice, we evaluate the performance of IAG- Student with varying statement numbers.
As shown in Figure 4 , IAG- Student achieves the best performance with the statement number between 5 and 7.
The performance drops when the sampling number is either too large or too small.
On the one side, a small sampling number makes the model prone to random errors.
On the other side, sampling too many statements inevitably introduce more noisy information that could mislead the prediction. Hence our choice of 5 knowledge statements makes a reasonable trade-off.
Conclusion
To tackle the problem that retrieval-based methods cannot provide sufficient knowledge for the generator to answer implicit reasoning questions, we propose a novel IAG framework that augments RAG with inductive knowledge elicited from language models.
We first design an inductive prompting method that enhances the factuality of knowledge elicited from GPT-3.
We further propose a sophisticated optimization scheme that trains a student inductor model via distillation and TailBack .
Our results suggest that IAG outperforms RAG in answering implicit reasoning questions.
Limitations
This work has several limitations.
First, IAG has evident advantages over RAG only for questions that cannot be readily answered by the retrieved documents. Otherwise, the performance boosts brought by providing inductive knowledge are less significant.
Second, the effectiveness of IAG- Student and the proposed distillation and TailBack optimization scheme has only been verified on the T5-Large architecture.
We leave it as our future work to experiment our methods with larger models and various backbones.
B.1 Comparison between Information Retrieval and knowledge Induction
Table 8 lists some cases where the retrieved documents fail to provide informative evidence whereas the inductive knowledge stands out.
Typical failures of retrieval-based approaches can be categorized into two occasions.
Firstly, when the knowledge required for answering the question is too scarce to be found, the retriever could fetch documents that hardly match the semantics of the question (e.g., Q3).
Secondly, when the question is too trivial and the answer is unlikely to be officially documented, the retrievals could contain specific cases that contradict the commonsense conclusion (e.g., brittle failure of steel for Q1 and unequal leg length for Q2).

Besides, the results on different setups of IAG-GPT suggest that, the relative contributions of the retrieval and the inductive knowledge can be different, depending on the tasks.
As shown in Table 1 , for CSQA2.0, higher scores are reported for retrieval only than for induction only, while the result is contrary for StrategyQA.
These results can be attributed to the fact that questions in StrategyQA (e.g., Would a cattle farmer be useful to a drum maker? ) require better reasoning ability to answer.

To fully verify the effectiveness of the inductive knowledge, we compare the performance of IAG based on two different settings: 1) using 10 retrievals and 2) using 5 retrievals and 5 knowledge statements. The results are shown in Table 7 . For CSQA2.0, using more retrievals yields a slightly better result, while for StrategyQA, introducing knowledge statements significantly boosts the performance. This finding is consistent with our previous conclusion that our method works better for reasoning-intensive QA tasks. For CSQA2.0, many commonsense questions can be solved by referring to plain-text records by retrieving methods. However, for StrategyQA, the answers are unlikely to be directly extracted from the retrieved documents but require reasoning over multiple documents to obtain. In such cases, introducing inductive knowledge can be very useful.

This is also the reason why the marginal performance gains are more significant for StrategyQA than for CSQA2.0 when inductive knowledge is offered besides the retrieved documents.
B.2 Comparison among Prompting Methods
Table 9 compares the reasoning traces of different prompting methods.
Although CoT is regarded a reliable prompting method by constructing intermediate reasoning steps, it’s also prone to errors that could occur at any point of the reasoning path.
For example, the sentence Thus, Christmas trees are not dissimilar to deciduous trees is a false deductive result based on the previous statements, leading to a false conclusion.
As shown in Table 2 , although enhancing CoT with self-consistency improves performance ( 71.5\rightarrow 73.3 ), the intrinsic problem identified above still plagues this approach.
Let alone the advantage that IAG can utilize retrieved documents as extra information for prediction (76.2), the ablated version that employs only the GPT-3 knowledge (75.6) still has a lead over CoT.
B.3 Effect of Inductor Optimization
We list some cases in Table 10 that compares the knowledge statements generated by the student inductor model at different optimization stage.
As shown, after TailBack training, the inductor produces knowledge statements that are more consistent.
Besides, we find that, although distillation training enables the inductor to grasp the inductive reasoning pattern, the generated knowledge still suffers from factual errors (e.g., #D3 for Q1), which is probably due to the limitation imposed by a small model size.
In comparison, further optimization via TailBack deviates the inductor from the pre-defined reasoning pattern, but the knowledge (e.g., #T1 for Q2) can better guide the generator in predicting the right answer.
Figure 1: Overview of our IAG framework. The inductor provides inductive knowledge for predicting the answer when the retrieved documents are less informative.; Overview of our IAG framework. The inductor provides inductive knowledge for predicting the answer when the retrieved documents are less informative.
Figure 2: Illustration of our proposed TailBack training scheme. The likelihood values of the ground-truth answer conditioned on different knowledge statements are used as feedback from the generator, which can be back-propagated to the inductor via differentiable beam scores.; Illustration of our proposed TailBack training scheme. The likelihood values of the ground-truth answer conditioned on different knowledge statements are used as feedback from the generator, which can be back-propagated to the inductor via differentiable beam scores.
Figure 3: Comparison of different warmup strategies. Scores are reported on the StrategyQA dev set.; Comparison of different warmup strategies. Scores are reported on the StrategyQA dev set.
Figure 4: Scores of IAG- Student on StrategyQA dev set with different numbers of knowledge statements.; Scores of IAG- Student on StrategyQA dev set with different numbers of knowledge statements.
Performance on two ODQA tasks. The first two columns report scores on CSQA2.0 dev set and StreategyQA test set respectively. The last two columns compare IAG with ChatGPT on a randomly held-out subset containing 50 examples for each task.
Comparison between IAG-GPT and CoT prompting. Scores are reported for the StrategyQA dev set.
Performance of IAG- Student at different stages on the dev sets of CSQA2.0 and StrategyQA.
Comparison between knowledge fusion and self-consistency.
The inductive prompting template used in the experiments.
The trivial prompting template used in the experiments.
Performance comparison between different settings on CSQA2.0 and StrategyQA dev sets.
A demonstration of knowledge elicited from GPT-3 for cases in the CSQA2.0 dev set. Text colored in red indicates factual errors or contradictions to the ground-truth answer, and green indicates supporting evidence.