Teaching Small Language Models to Reason for Knowledge-Intensive Multi-Hop Question Answering (2024)

Abstract: Large Language Models (LLMs) can teach small language models (SLMs) to solve complex reasoning tasks (e.g., mathematical question answering) by Chain-of-thought Distillation (CoTD). Specifically, CoTD fine-tunes SLMs by utilizing rationales generated from LLMs such as ChatGPT. However, CoTD has certain limitations that make it unsuitable for knowledge-intensive multi-hop question answering: 1) SLMs have a very limited capacity in memorizing required knowledge compared to LLMs. 2) SLMs do not possess the same powerful integrated abilities in question understanding and knowledge reasoning as LLMs. To address the above limitations, we introduce Decompose-and-Response Distillation (D&R Distillation), which distills two student models, namely Decomposer and Responser separately. The two models solve a knowledgeintensive multi-hop question through an interactive process of asking and answering subquestions. Our method offers two advantages: 1) SLMs have the capability to access external knowledge to address subquestions, which provides more comprehensive knowledge for multi-hop questions. 2) By employing simpler subquestions instead of complex CoT reasoning, SLMs effectively mitigate task complexity and decrease data prerequisites. Experimental results on three knowledge-intensive multi-hop question answering datasets demonstrate that D&R Distillation can surpass previous CoTD methods, even with much less training data1.

heading: 1 Introduction
text: Large language models are capable of answering complex questions (e.g., mathematical questions)
*Corresponding author 1Our code will be available at https://github.com/
Xiang-Li-oss/D-R-Distillation
by generating step-by-step natural language reasoning paths, namely Chains-of-thoughts (CoTs) (Wei et al., 2022). However, the ability to solve complex reasoning tasks through CoT prompting is considered an emergence that appears in very large models with at least tens of billions of parameters (Wei et al., 2022), such as PaLM of 540B (Chowdhery et al., 2022), GPT-3 of 175B (Brown et al., 2020), and LLaMA of 70B (Touvron et al., 2023).
Recent works have proposed to transfer the rea-
7804
soning ability of large models to small language models (SLMs) through Chain-of-thought Distillation (CoTD) (Ho et al., 2023; Magister et al., 2023; Li et al., 2023a). Specifically, as shown in the upper part of Figure 1, they leverage the LLM (e.g., ChatGPT) to generate high-quality rationales and finetune a SLM with rationale-augmented questionanswer pairs. CoTD has successfully enhanced SLMs’ reasoning ability on many reasoning tasks, such as arithmetic reasoning (Cobbe et al., 2021), commonsense reasoning (Talmor et al., 2019), and symbolic reasoning (Wei et al., 2022).
However, previous CoTD works did not effectively address knowledge-intensive reasoning tasks such as multi-hop question answering (Petroni et al., 2021; Trivedi et al., 2023). Unlike arithmetic reasoning and commonsense reasoning, knowledge-intensive reasoning tasks pose greater challenges due to their requirement for both background knowledge and the ability to perform multistep reasoning. CoTD has two limitations that render it unsuitable for teaching SLM to reason over knowledge-intensive multi-hop question answering.
1) Knowledge Memorization Gap between LLMs and SLMs. Unlike LLMs, which store vast amounts of knowledge within their parameters, SLMs are limited in their capacity to memorize the necessary knowledge to solve the tasks due to their small number of parameters. Besides, simply augmenting SLM with a one-step retrievalaugmentation strategy (Kang et al., 2023; Zhang et al., 2023) is also suboptimal for multi-hop questions. For such questions, relevant knowledge often needs to be retrieved after intermediate reasoning has concluded, as it may not be explicitly mentioned in the question. For example, consider the question illustrated in Figure 1, one must first infer that the eastern sector of Colorado extends into the High Plains, and then perform further retrieval to obtain evidence pointing to the evaluation range.
2) Difficulty in Distilling Integrated Subtasks. In contrast to arithmetic reasoning, which typically involves applying predefined formulas or algorithms, or commonsense reasoning, which relies on general knowledge and intuition. Solving a knowledge-intensive multi-hop question via chainof-thought reasoning potentially involves a collection of multiple subtasks, including complex question decomposition, knowledge association, and knowledge reasoning (Zheng et al., 2023). How-
ever, it is highly challenging for an individual SLM to simultaneously acquire all these integrated capabilities, which leads to the CoTD methods requiring more training data and being inefficient.
To address the aforementioned limitations, motivated by question decomposition for answering complex questions (Han et al., 2023; Press et al., 2023), we propose a novel method to teach SLMs to reason for knowledge-intensive multi-hop questions, namely Deompose-and-Response Distillation (D&R Distillation, as shown in Figure 1). Specifically, we propose to prompt LLM in a SelfAsk-Self-Ans strategy by iteratively asking subquestions and responding with intermediate answers. Then we separately distill two student models, namely Decomposer and Responser. The Deomposer is responsible for asking subquestions and determining the final answer based on current interaction history. The Responser is responsible for answering subquestions by leveraging relevant background knowledge obtained from an external knowledge base. By formatting the reasoning process as a sequence of generating subquestions and intermediate answers, these two student models effectively address knowledge-intensive multi-hop questions within an interactive framework.
Compared with previous Chain-of-thought Distillation methods, our method offers two notable advantages: 1) By reasoning in an interactive manner, our method allows student models to utilize external knowledge with each retrieval focusing on a subquestion. Compared to previous works relying solely on parameter knowledge or one-step retrieval augmentation (Ho et al., 2023; Kang et al., 2023), our method provides a more comprehensive collection of relevant knowledge required to answer multi-hop questions. 2) We transform the process of solving a reasoning question into two interrelated and decoupled subtasks: decomposing the complex question and solving a series of simpler subquestions. D&R Distillation effectively reduces the overall task difficulty while significantly reducing the amount of data required for distillation.
We evaluate the effectiveness of our method on three knowledge-intensive multi-hop question answering datasets: HotpotQA, StrategyQA, and 2WikiMultiHopQA. Experimental results demonstrate that D&R distillation significantly improves the knowledge-intensive reasoning ability of SLMs with approximately 1/10 of the full training data. Notably, our method with two 220M SLMs (T5base) outperforms Chain-of-thought Prompting
with an 11B (50 times larger) LLM (Flan-T5-XXL) on HotpotQA and 2WikiMultiHopQA.

heading: 2 Related Work
text: Chain-of-Thought prompting (Wei et al., 2022) significantly enhances the reasoning capacities of large language models by augmenting few-shot examples with detailed reasoning steps. Recent works have further refined CoT through verification (Li et al., 2023b), question decomposition (Zhou et al., 2023), and path sampling (Wang et al., 2023; Yao et al., 2023). However, these aforementioned studies primarily concentrate on enhancing the reasoning capabilities of LLMs, neglecting the necessity to improve the reasoning abilities of smaller language models (<1B).
Chain-of-thought Distillation have been proposed to distill the CoT reasoning ability of LLMs into SLMs (Ho et al., 2023; Fu et al., 2023; Magister et al., 2023; Hsieh et al., 2023), because the CoT reasoning ability is considered as an emergent ability which enables LLM to generate intermediate reasoning steps with CoT prompting (Wei et al., 2022) (e.g. Let’s think step by step). To augment Chain-of-thought Distillation (CoTD) with external knowledge, (Kang et al., 2023) augment SLMs with documents retrieved by a one-step retriever from the external knowledge base. However, CoTD is less effective for knowledge-intensive multi-hop question answering tasks (Petroni et al., 2021), where both factual knowledge and multi-hop reasoning are important to generate accurate rationale. In this paper, we propose to distill two student models and solve a knowledge-intensive multi-hop question by facilitating an interactive process of asking and answering subquestions between the two student models.
Question Decomposition (Kalyanpur et al., 2012; Patel et al., 2022) has long been a crucial technique for understanding and solving complex questions. Recent works also utilize question decomposition to improve the reasoning ability of LLMs. (Zhou et al., 2023) enhances the CoT reasoning ability of LLMs by decomposing questions into subquestions and sequentially solving subquestions. (Press et al., 2023) explicitly asks LLM itself follow-up subquestions before answering the original question and answers subquestions with an external search engine. (Shridhar et al., 2023) learns a semantic decomposition of the original question
into a sequence of subquestions and uses it to train two models designated for question decomposition and resolution. Unlike the aforementioned works, we focus on teaching small language models to reasoning for knowledge-intensive multi-hop questions with LLM generations. We achieve this by distilling two student models to interactively ask and answer subquestions.

heading: 3 Method
text: In this section, we provide a detailed description of our method. As illustrated in Figure 2, D&R Distillation can be divided into three stages:
1) Self-Ask-Self-Ans Prompting: We prompt a very large language model (e.g., ChatGPT) to generate D&R Distillation samples, preparing datasets for training student models.
2) Decomposer and Responser Training: We distill two student models (e.g., T5) with D&R Distillation samples obtained by stage 1).
3) Decomposer and Responser Interaction: The Decomposer and the Responser address a knowledge-intensive multi-hop question through an interactive process of generating subquestions and obtaining intermediate answers.
3.1 Self-Ask-Self-Ans Prompting In this stage, a teacher model (LLM) is prompted with Self-Ask-Self-Ans prompting to generate D&R Distillation samples2. Specifically, the teacher model solves a knowledge-intensive multi-hop question by iteratively asking itself subquestions and providing intermediate answers. Consider a standard sample Si consisting of a question qi and its golden answer ai. The teacher model serves as a Decomposer and a Responser alternatively. At the k-th step, when serving as a Decomposer, the teacher model decide to continue asking a subquestion ski or predicting the final answer a k i based on interaction history:
H =< qi, s 1 i , r 1 i , ..., s k−1 i , r k−1 i >
where sti and r t i are the subquestion and the intermediate answer of the t-th step. When serving as a Responser, the teacher model answers the subquestion ski proposed before with retrieved passages:
P ki = topK(R(p|ski ;D),K)
rki = LLM(P k i , s k i )
2Prompting examples for the teacher model can be found in Appendix B
where R is a retriever and D is a knowledge base (e.g., Wikipedia). Once the teacher model decide to predict the final answer aki , we obtain a D&R Distillation sample (qi, s1i , r 1 i , ..., s k−1 i , r k−1 i , a k i ). Moreover, to control the quality of generated samples, we filter generated D&R Distillation samples by comparing the final prediction aki of the teacher model with the ground truth ai. More detailed filter criteria can be found in Appendix A.
3.2 Decomposer and Responser Training
After acquiring D&R Distillation samples, we use them to fine-tune two small student models, namely the Decomposer pdθ and the Responser prϕ with trainable parameters θ and ϕ respectively. Specifically, consider a D&R Distillation sample (qi, s 1 i , r 1 i , ..., s k−1 i , r k−1 i , a k i ), for the Decomposer, we minimize the negative log-likelihood of the sequence of subquestions sji (j = 1, 2, ..., k − 1) and the final answer aki :
LD(θ) = − N∑
i=1
k∑
j=1
log pdθ(o j i |H)
(oji = a j i if j = k else s j i )
(1)
where H represents the interaction history before j-th step:
H =< qi, s 1 i , r 1 i , ..., s j−1 i , r j−1 i >
For the Responser, we minimize the negative loglikelihood of the sequence of intermediate answer rji with augmented external knowledge:
P ji = topK(R(p|s j i ;D),K)
LR(ϕ) = − N∑
i=1
k∑
j=1
log prϕ(r j i |s j i , P j i )
(2)
where R is the same retriever in 3.1.
3.3 Decomposer and Responser Interaction This section describes the behavior of two student models in the inference stage. After the aforementioned two stages, the Decomposer and the Responser work interactively to jointly solve a knowledge-intensive multi-hop question. As shown in Algorithm 1, we initiate with feeding the initial question to the Decomposer, at the j-th step, the Decomposer decides whether to ask another subquestion or predict the final answer based on current interaction history H . If the generation of the Decomposer is another subquestion, then the Responser retrieves related knowledge from a
Algorithm 1 Inference of D&R Distillation 1: Initialization: H = {qi}, MAXSTEP ← T ,
j ← 0, pdθ , prϕ, R, D, K 2: repeat 3: oji = argmaxop d θ(o|H) 4: if oji is subquestion then 5: P ji = topK(R(p|s j i ;D),K) 6: rji = argmaxrp r ϕ(r|o j i , P j i ) 7: H .append(oji , r j i ) 8: end if 9: if oji is final answer then
10: break 11: end if 12: j ← j + 1 13: until j =MAXSTEP Output: final answer oji
knowledge base and generates a response to the subquestion. Otherwise, if the generation of the Decomposer is the final answer, the interaction terminates and returns the final answer.

heading: 4 Experiments

heading: 4.1 Datasets
text: We evaluate our method on three knowledgeintensive multi-hop question answering datasets in the open-domain setting: HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (Ho et al., 2020), and StrategyQA (Geva et al., 2021). In contrast to previous works (Ho et al., 2023) of fine-tuning with the entire training set, we only fine-tune our model with 8800 instances (1/10 of the full training data) for HotpotQA, 16000 instances (1/10 of the full training data) for 2WikiMultiHopQA, and 1200 (1/2 of the full training data) instances for StrategyQA, eliminating the need for generating a large number of rationales with LLMs.

heading: 4.2 Teacher and Student Models
text: For teacher models, we use GPT3.5 (Brown et al., 2020) provided by the OpenAI API. Unless otherwise stated, we use gpt3.5-turbo-instruct as the teacher model. For student models, we adopt T5-{Small, Base, Large} (Raffel et al., 2020).

heading: 4.3 Baseline Methods
text: We provide a comparison of D&R Distillation (ours) with four baseline methods: Fine-tuning directly fine-tunes a student model to generate an answer given only a question (Petroni et al., 2021).
CoT Distillation finetunes a student model with LLM-generated rationales, which is a typical approach for enhancing the reasoning capabilities of SLMs (Ho et al., 2023). The above baselines measure the capability of a small language model to solve knowledge-intensive multi-hop question answering relying only on parameter knowledge but without any external knowledge.
Retrieval-Augmented Fine-tuning appends retrieved passages along with the question at both training and inference time (Petroni et al., 2021). Retrieval-augmented CoT Distillation augments CoT Distillation with retrieved passages for both teacher and student models (Kang et al., 2023). The above two baselines help us to investigate the impact of incorporating external knowledge.

heading: 4.4 Implementation Details
text: We fine-tune student models for a maximum of 20 epochs with Pytorch-Lightning library3, setting the batch size at 16 and the learning rate at 3e− 4.
For Retrieval-augmented methods, we use Wikipedia as the external knowledge base. For a fair comparison, we use the sparse retrieval method BM25 as the retriever provided by Pyserini library 4 for all baseline methods and our method. See Appendix A for more detail.

heading: 4.5 Experimental Results
text: In this section, we present the knowledge-intensive reasoning performance of our D&R Distillation. We compare our method with various baselines across different model sizes.
As shown in Table 1, the improvement of Chainof-thought Distillation (CoT Distillation) compared to Fine-tuning is quite limited, and in some cases, even a performance decline has been observed. For example, T5-base exhibits a mere 0.9% (32.5%- 31.6%) increase in Answer F1 on 2WikiMultiHopQA whereas it encounters a 0.4% (19.3%- 19.7%) drop in Answer F1 on HotpotQA. This phenomenon can be highly attributed to the lack of background knowledge. Although CoT Distillation trains SLMs with augmentation of intermediate reasoning steps, it remains a challenge for SLMs to effectively reason without the necessary background knowledge.
The application of retrieval augmentation benefits both Fine-tuning and CoT Distillation. For example, the utilization of retrieval augmentation
3https://lightning.ai 4https://github.com/castorini/pyserini
leads to a noteworthy improvement in the performance of T5-base. It enhances the Answer F1 of HotpotQA from 19.3% to 22.2% and increases the Answer accuracy of StrategyQA from 55.3% to 56.6%. However, augmenting CoT Distillation with a one-step retriever alone can not achieve comparable results to our method except for the StrategyQA dataset with T5-large. We attribute this discrepancy to the nature of the StrategyQA dataset, which consists of relatively easier yes/no questions. Therefore, it becomes easier for a model to find shortcuts to reach the final answer.
In contrast, D&R Distillation improves the knowledge-intensive reasoning ability of SLMs by a large margin and surpasses all baseline methods with student models of different sizes. Moreover, the performance gap between D&R Distillation and Fine-tuning baseline enlarges as the number of parameters of the student model increases. With T5-large, D&R Distillation achieves an Answer F1 gain of 8.3% and 6.5% over Fine-tuning on HotpotQA and 2WikiHotpotQA respectively.
Furthermore, it is noteworthy that our approach is trained using a significantly smaller fraction of
the data compared to the baseline methods. For the larger-scale HotpotQA and 2WikiMultiHopQA datasets, we utilize only 1/10 of the training data, while for the smaller-scale StrategyQA dataset, we use only 1/2 of the training data. The above findings highlight the significant advantages of our method in terms of both performance and efficiency. Unlike existing (Retrieval-augmented) CoT Distillation methods, which heavily rely on extensive CoT annotations but struggle to effectively enhance the model’s knowledge-intensive reasoning capabilities, our approach achieves superior performance, despite utilizing only a small fraction of data.

heading: 4.6 Analysis
text: Efficiency on Model Size and Training Data To validate the efficiency of our D&R Distillation method in terms of model size and training data, we measure the Answer F1 on HotpotQA and 2WikiMultiHopQA varying model parameters and the Answer F1 on HotpotQA varying the number of training data. As shown in Figure 3a, D&R Distillation consistently outperforms the CoTD and RACoTD baselines varying different model sizes with
only 1/10 of the entire training dataset. Notably, on the HotpotQA dataset, D&R Distillation with two 60M student models achieves higher Answer F1 than CoTD with a 700M student model, whether enhanced with Retrieval augmentation. Moreover, D&R Distillation with two 220M student models outperforms the 11B LLM (FLAN-T5-XXL) incontext learning baseline. This observation shows a significant practical advantage of our approach in resource-restricted settings since the SLM with D&R Distillation requires significantly less computational cost yet it outperforms the LLM.
As shown in Figure 3b, the proposed D&R Distillation method can successfully transfer the knowledge-intensive reasoning ability, using only a small number of training data. Specifically, with 10% of the training data, D&R Distillation signif-
icantly outperforms the same model distilled by RA-CoTD on the full data by 5.7% on answer F1. With 2.5% of the training data, D&R Distillation still outperforms RA-CoTD on the full data. This indicates that we can potentially further increase the effectiveness of D&R Distillation with more training data.
Evaluation on Knowledge Acquisition To evaluate the effectiveness of D&R Distillation on retrieving relevant knowledge for knowledgeintensive multi-hop question answering, we compare the retrieval recall of our method with one-step retriever (OneR) adopted in Retrieval-augmented
highlight supporting facts in the passages as yellow.
baseline methods. As shown in Figure 4, our method achieved significantly higher recall compared to OneR. Particularly, D&R Distillation demonstrates a remarkable 20.6% superiority in recall over OneR on the 2WikiMultiHopQA dataset. This indicates that by decomposing and retrieving based on subquestions iteratively, D&R Distillation obtains a more sufficient set of knowledge to answer knowledge-intensive multi-hop questions.
Ablation Study We conduct an ablation study to demonstrate the effectiveness of two designs in our method: 1) incorporating multi-step retrieval based on subquestions and 2) interaction process between Decomposer and Responser. For 1), we disable the retriever and do not provide retrieved passages for Responser, denoted as w/o retriever. For 2) we train Decomposer to output all subquestions at once and train the Responser to output all intermediate answers, as well as the final answer at once, denoted as w/o interaction. We then compare the Answer F1 of the two ablation settings with our original design and the CoT Distillation (CoTD) baseline. As shown in Figure 5, both of these designs are crucial for our method, as the absence of either one would result in performance degradation. On the other hand, the performance without either of these designs still surpasses that of CoTD, demonstrating their strength. The performance decline becomes even more pronounced when the retriever is removed (w/o retriever), further confirming the crucial role of background knowledge for knowledge-intensive multi-hop reasoning.
Case Study In Table 2, we provide two examples from the HotpotQA dataset comparing the output generated by our D&R Distillation against the rationale by the baseline method Retrievalaugmented CoT Distillation (RA-CoTD). For the first question, RA-CoTD fails to retrieve a passage about Crime and Punishment, as a result, it mistakenly generates the hallucination that "Crime and Punishment" was first published in 2002. For the second question, RA-CoTD successfully retrieved the necessary knowledge for answering the question, however, it fails to perform correct reasoning by mistakenly assuming that Annie Morton (born in 1970) is older than Terry Richardson (born in 1965).
In contrast, D&R Distillation successfully retrieves a passage about Crime and Punishment by first generating subquestion When was Crime and Punishment first published and retrieving based on the subquestion. Also, D&R Distillation performs the correct reasoning by predicting that Terry Richardson is older. These examples highlight the effectiveness of our D&R Distillation method for reasoning interactively with adequately acquired relevant knowledge, which leads to a notably improved performance for knowledgeintensive multi-hop questions.

heading: 5 Conclusion
text: In this paper, we proposed Decompose-andResponse Distillation (D&R Distillation) which enhances the reasoning capabilities of small language models (SLMs) on knowledge-intensive multi-hop question answering. Our approach involves dis-
tilling two student models separately, with one student model focusing on decomposing subquestions and another student model focusing on answering subquestions with retrieved background knowledge. Through extensive experiments, we showed that D&R Distillation outperforms previous Chain-of-thought Distillation approaches with much less training data.
Limitations
We conduct experiments on three knowledgeintensive multi-hop question-answering datasets, demonstrating the effectiveness of D&R Distillation. However, our method is specially designed for knowledge-intensive reasoning tasks. This limitation poses a constraint on the wider applicability of our method. We plan to extend D&R Distillation to a wider range of reasoning tasks in the future. On the other hand, due to limitations in computational resources, we were unable to conduct experiments on larger-scale language models (> 1B). We will further explore the performance of D&R Distillation on larger-scale language models in future research.
Ethics Statement
The proposed method has no obvious potential risks. All the scientific artifacts used/created are properly cited/licensed, and the usage is consistent with their intended use. All the data used in this work contains no private information.
Acknowledge
This work was supported by the Strategic Priority Research Program of Chinese Academy of Sciences (No. XDA27020203) and the National Natural Science Foundation of China (No. 62376270, No. 62276264) and OPPO Research Fund.

heading: B Prompts
text: Prompting examples for the three datasets can be found on Table 3, Table 4, and Table 5.

heading: Future Work
text: We plan to extend D&R Distillation to a wider range of reasoning tasks in the future. On the other hand, due to limitations in computational resources, we were unable to conduct experiments on larger-scale language models (> 1B). We will further explore the performance of D&R Distillation on larger-scale language models in future research