Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks
Authors: Minki Kang
Seanie Lee
Jinheon Baek
Kenji Kawaguchi
Sung Ju Hwang
Venue: NeurIPS 2023 poster (2023)
Keywords: language model
distillation
reasoning
knowledge augmentation
TL;DR: reasoning distillation with knowledge augmentation for small language models

Abstract: Large Language Models (LLMs) have shown promising performance in knowledge-intensive reasoning tasks that require a compound understanding of knowledge. 
However, deployment of the LLMs in real-world applications can be challenging due to their high computational requirements and concerns on data privacy.
Previous studies have focused on building task-specific small Language Models (LMs) by fine-tuning them with labeled data or distilling LLMs. However, these approaches are ill-suited for knowledge-intensive reasoning tasks due to the limited capacity of small LMs in memorizing the knowledge required.
Motivated by our theoretical analysis on memorization, we propose Knowledge-Augmented Reasoning Distillation (KARD), a novel method that fine-tunes small LMs to generate rationales obtained from LLMs with augmented knowledge retrieved from an external knowledge base. Moreover, we further propose a neural reranker to obtain documents relevant to rationale generation. We empirically show that KARD significantly improves the performance of small T5 and GPT models on the challenging knowledge-intensive reasoning datasets, namely MedQA-USMLE, StrategyQA, and OpenbookQA.
Notably, our method makes the 250M T5 models achieve superior performance against the fine-tuned 3B models, having 12 times larger parameters, on both MedQA-USMLE and StrategyQA benchmarks.

Knovledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Inteniiee Tasks
Minki Kang 1:2 Seanie Lee?, Jinheon Back?, Kenji Kawaguchi Sung Ju Hwang KRAFTONN, KKIST, "National University of Singapore, 'DeepAuto.ai (22xc1133, lsnf aen02,, j jinheon. banke) @kaist. ac.kr, kenji@comm nus edu, sg, sjhwang620kaist.o aattaaa... ac.kr
Abstract
Large Language Models (LLMs) have shown promising performance in knowledge- intensive reasoning tasks that require a compound understanding of knowledge. However, deployment of the LLMs in real-world applications can be challeng- ing due to their high computational requirements and concerns on data privacy. Previous studies have focused on building task-specific small Language Models (LMs) by fine-tuning them with labeled data or distilling LLMs. However, these approaches ill-suited for knowledge-intensin ited capacity of small LMS in memorizing the knowledge required. Motivated by our theoretical analysis on memorization, we propose Knowledge-Augmennte Reasoning Distillation (KARD), novel method that fine-tunes small LMs to generate rationales obtained from LLMs with augmented knowledge retrieved from an external knowledge base. Moreover, we further propose a neural reranker obtain documents relevant rationale generation. We empirically show that KARD significantly improves the performance of small T5 and GPT models on the challenging knowledge-intensinn reasoning datasets, namely MedQA-USMIEE Strategy QA, and OpenbookQA. Notably, our method makes the 250M T5 models achieve superior performance against the fine-tuned 3B models, having times larger parameters, on both McdQA-USMLE and Strategy benchmarks.
reasoning tasks due to the lim-
Introduction
Large Language Models (LLMs) [5, 8] have excelled at various tasks across diverse domains with in-context learning. Recently, scaling up the number of parameters of LLMs has been shown to significantly improve their knowledge encoding and reasoning capability [54, 24]. Moreover, such LLMs have achieved remarkable performance on knowledge-intensivy tasks in professional domains which are highly challenging, since they require considerable depth of domain knowledge and reasoning [34, 48). For example, in Figure top, answering a medical question requires both domain knowledge and reasoning ability. The LLM should understand that the patient likely has ALS based on the symptoms and recognize SODI is the main cause of motor neuron diseases. Furthermore, it needs to reason over the knowledge that mutation in SODI is highly associated with the symptoms.
Despite its effectiiveness, deploying LLMs can still be challenging, especially in real-world applica- tions. Firstly, utilizing LLMs make predictions is computationally expensive. It requires 326GB GPU memory to load the GPT3-175B model [11]. Moreover, deployment of the LLM potentially poses risk of privacy leakage since most of the production-gren
LLMSs [5, 44, 8, 43] operate in black-box manner. That is, users cannot access the parameters of LLMS but only their output via some Application Programming Interfaces. Consequently, the need for white-box Small Language Models tailored to address problems requiring domain-specific knowledge will continue to gain
WWrk done AITRICS. Code is available
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

history uppe
forcarm atreshy and depreszed refilsess while his right lower oxtrensty hypartonic with positive Boblaski zigh, mhzch of the following most liely asste infedd with the cass of This pwt lent's symphres? A) BiA--i hpp dt) HEA-0R2 haplotype C) Nutation is 5001 BJ Viral infection
aptions, discases, A viral lnfection can case peripheral neurspathy, but the syeptons described suggest centrall nervous System disordar,
Rationate:
spepizes suppest
Reasoning Dicillation
KARD gusseg!
KARD (xioeav)
Relevant knowletge fretrieved with fationale)
Questioon tion
Question
Quastios
Retriever
Knowledie
[Superocide dicrutase] titaiiinss
Inpert
KnowLedge
[5001] faeiliet anyotrophic Lakeral sclerosis fore of sisease).
Irgut
?
Sea
Sew LM
Smalt LM
Irrelevant Ksowledge with 0uest100)
Fer Beranker
1 Rationale
Label
Tabed
with A artipens wore färge
(HLA Al-hi hep ypel A1-n3 verotype was wexer
Sewinl S0t
Rationale
Rationala
besing described. Coea)
Figure I: Concept. An example of knowledge-intensinn reasoning task (medical QA [23]) on the top. On the bottom, we provide the conceptual illustration of our KARD, compared existing reasoning distillation. On the right, we provide examples of passages retrieved with rationale and question from the external KB.
prominence. To tackle the above challenges of deploying models, previous works [33, 17, 38. 12, 18] have proposed transfer the reasoning ability of large models to small models through reasoning distillation (See Figure left). In particular, they leverage the L.M to generate high-quality rationales and fine-tune a small LM to generate the rationale obtained from the LLM. This reasoning distilla- tion improves the performance of small LMs on tasks that require complex reasoning ability (e.g.. arithmetic and symbolic reasoning [10, 55]). Based on this observation, we pose a research question: "Is it possible to transfer both the domain knowledge and reasoning ability of LLMs through reasoning distillation, for tasks requiring specific knowledge to reason for answering a question?"
Existing reasoning distillation is suboptimal to solve such knowledge-intensive intensive reasoning tasks since small, distilled LMs are limited in their capacity to memorize the knowledge that is necessary to solve the tasks due to the small number of parameters. This motivates us to develop a method that distills the reasoning ability of L.LMs into smaller LMs while injecting the specific task-relevar knowledge. Specifically, we augment small LM with the knowledge retrieved from an external Knowledge Base (KB) as a non non-parametric memory, and we theoretically show that the non- con-aaraettric memory can reduce the number of bits to memorize training data for performing well.
Based on this intuition and the theoretical analysis, we propose Knowiedge-Augment Distillation (KARD) which enables to transfer the reasoning ability of an LLM to small LM while injecting the knowledge, for wowwledgeinttessing reasoning tasks. Specifically, we utilize a retriever [46] to obtain passages containing relevant knowledge for generating a rationale from an external knowledge base (c.g.. Wikipedia). We then fine-tune the small LM to generate the rationale, obtained from the LLM,, based on the question and the retrieved document, and predict the answer.
Reasoning
During training, using rationale as query helps retrieve pertinent knowledge for generating rationales. However, during inference, relying on the question as a query may result in poor retrieval. As shown in Figure/ the passage retrieved with the question is not relevant to generating the rationale. To mitigate the issue, we introduce a neural reranker to prioritize passages useful for rationale generation, ensuring retrieval of relevant documents even with the question the query.
To verify the efficacy of KARD, we empirically show that significantly improves the performance of small LMs (OPT [20, 59] and T5 [45, 53]) on medical Question Answering (QA) (MedQA- USMLE [23]), multi-step factual QA (StrategyQA [14)), and commonsense reasoining (Open- bookQA (39]) datasets compared to few-shot in-context learning, fine-tuning, and reasoning distilla- tion without knowledge augmentation. Also, our extensive analyses demonstrate that our KARD is efficient in terms of both the training data and the model size. Specifically, KARD with 250M models achieves higher accuracy than the fine-tuned 3B models, and KARD outperforms the fine-tuning only with quarter of the full training data in 780M models.
Our findings and contributions are as follows:
We demonstrate that fine-tuning small LMs to generate rationales from large LMs is insufficient for Enowledge-intent inngge-veteeivve reasoning tasks and non-parametric external knowledge base plays erucial role in complementing the lack of knowledge in small LMs.
Moreover, we address the limitations of the existing retriever method by introducing a reranker, in order to obtain pertinent passages for generating rationales in knowledge-intenss reasoning tasks.

---

In widely-used medical, multi-step factual, and commonsense QA benchmark datasets, we empiri- cally show that the proposed KARD significantly improves the performance of small LMs.
2
Related Works
Large Language Models Large Language Models (LLMs) have shown impressive capabilities across various tasks. One of their notable strengths is their ability to memorize knowledge and leverage that knowledge to solve knowledge-intensive reasoning tasks. For example, LLMs like GPT-3.5 [44], Med-PaLM [48], ChatGPT [29], and GPT-4 [43] have shown the promising performance on the challenging medical question answering task, the United States Medical Licensing Examination (USMLE) [23], even surpassing the passing score by a large margin [41]. However, deploying LLMs in offfine and privacy-sensittee environments is still challenging since most of these models are in black-box (accessible via APIs), and computationally expensive. Thus, we need alternative solutions that can leverage the capabilities of LLMs for knowledge-ing inttegsive reasoning tasks.
Reasoning Distillation from LLMs Recent works [33, 17, 38, 12, 18] have attempted to distill the reasoning ability of LLMs into small LMs, where the reasoning ability is an emergent property which enables LLMS to perform better in reasoning tasks through Chain-of- Thought (CoT) prompting (e.g., Let's 's think step-by-seep) [28, 55J. Unlike arithmetic symbolic reasoning tasks, however, previous works [33, 17] have shown that reasoning distillation less effective for knowledge-inton intteeiiiin reasoning tasks [14] where factual knowledge is important to generate accurate rationale. Therefore, we augment small LMs with documents retrieved from the external knowledge base so that the models can leverage knowledge generate better rationales that lead to correct answers.
Knowledge-Augmenttee LMs showledge-augmented LMs have utilized an external Knowledge Base (KB) to supplement their intrinsic knowledge [16, 32, 3, 22, 58]. One common approach to incorporate external knowledge is by retrieving relevant passages from a KB, such as Wikipedia, based on the input query [7]. Retrieving the correct evidence is crucial to generate accurate answers and factually grounded rationales. However, previous works usually have not explored the use of knowledge-augmennt
LMs for tasks that require complex reasoning over knowledge. Recently, BehnamGhader et al. [1] examined the reasoning ability of cxisting retrieval-augmented LMs and found that the existing retriever [25] is insufficient for retrieving relevant passages to solve the knowledge-intensive reasoning tasks. To address this limitation, we propose a re-ranker for rationale generation that prioritizes passages relevant the rationale generated by LLMs given the query. This approach can be seen as a form of knowledge distillation for the retriever, as we use the rationale to guide the reranker to retrieve more relevant passages for reasoning, instead of using plain queries.
3 Motivation: Effect of Knowledge-Augmentann
on Memorization
Large language models are known memorize its training data [6, 49] and the memorization capacity is proven to increase as the size of the model increases [27, 57]. The previous work [4] showed that the memorization of training data is indeed necessary to perform well in a language problem. These results suggest that the reasoning distillation with a small language model (without knowledge augmentation) will degrade the performance because of (I) the incapability of memorizing training data and (2) the necessity of the memorization to perform well. In this section, we demonstrate that using an external Knowledge Base (KB) as a parammemttrric memory with a retriever reduces the amount of the memorization needed to perform well and thus allows us to small models.
3.1 Background without
We adopt the exact same problem setting used in Brown et al. [4]. A task distribution P ~ g is drawn from meta-distrin distribittiiitonn
Given a P. the training dataset = ((Z,, Y.)); and the test sample (Z, Y) drawn X p&n and (Z, Y) P. Here, Z is the input (i.e., the sequence of symbols) and Y is the label (i.e., the next symbol to be predicted). The overall error of a learning algorithm A on the meta-distribution q with sample size defined by
eIT g. (A)=
Pr X= Pwq. (2
(M(Z) + Y where M A(X)).
Given y and n, there exists an optimal learner AOrT that minimizes this overall error, which will be used as our reference. We adopt the abstracted language problem, i.e., the next-symbol prediction problem with N reference strings {c, J.N where ej Uniform ((0, 1)d), considered in the main
3

---

text of Brown et al. [4] with no symbol corruption (see [4] or Appendix A.1 for the details). Under this setting, Brown et al. [4] proved that any algorithm A needs to memorize the nd bits of training data to achieve e-suboptimality where 7 denotes the mutual information:
Theorem 1 (Brown et al. [4]). Let N n. Then, any learning algorithm A that satisfies e (A) S err, (AooT)(AOPT) + f = o(1) also satisfies A(X) = Q(nd).
3.2 Memorization with Knowledge-Angmeentttio
In Theorem 1, corresponds to the size of KB. Thus, it shows that if the size of KB is small, then a small model can just memorize all KB by memorizing ((nd) information to perform well. However, if the size of KB is large, then a small model cannot memorize ((nd) information and hence the performance is expected to drop significantly when replacing large model with small model. In this subsection, we show that saweldeg--aggmentttoonn reduces the memorization requirement of ((nd) bits to that of On log2(N R)) bits, allowing the use of small models.
We consider an inference algorithm yp that uses a KB with a non-parametric retriever as follows: erry q. (A)
Pr (, S) + Y where M = A(X)).
P~g.X~~Pn (2,Y)~P
An inference algorithm yp has learnable parameters and makes prediction based on both the result of learning algorithm M = A(XX) and a KB denoted by S, which is defined as follows. Given a task instance / ~ q, we choose a KB such that |S] = N + R and {e, J. S S where I is the number of extra references that are irreverent to this task P; i.e., R = 0 in the best scenario.
Theorem 2 shows that the wwwwwlldg--uggeetten e-suboptimality, from the nd to min( (N, n ) bits, under the same problem setting as Theorem 1: Theorem 2. There exists a pair of inference and learning algorithms (4,, A) such that for any > 0. err q.n (A) S CIT, (AoPT) + C. and I(X; A(X) O(min(N, n)m) where m log2((1
reduces the amount of memorization to achieve
[--1 (N+R)* S 1ogg(
(N+R)*
2r
2r
With n N and e = o(1), we have I(X; A(X) P) ((, n)m) = In (NV + R)) (see Appendix A.2 for proof). Thus, it shows that knowledge augmentation
knowledge--ugmmenttiin allows the reduction from the
nd bits to n log2(N V + R) bits for the amount of memorization needed to perform well.
4 Knowledge-Augment
Reasoning Distillation
We propose Bhowledge-Augmennee Reasoning Distillation (KARD), which consists of two learning processes: (1) reasoning distillation where we leverage Large Language Models (LLMs) to generate a rationale with black-box APIs and then fine-tune small models to generate both rationale and answer given a question and knowledge, in which the knowledge is retrieved from Knowledge Base (KB) with the rationale as a query; (2) reranker training retrieve relevant passages for the question as query at the inference time, for generating effective rationales. Our approach is illustrated in Figure 2.
4.1 Teach Small Models Generate Rationales with External Knowledge
Rationale Generation with LLMs In our problem setup, we assume that training dataset ((x,, v))) for the target task is given, where x; is input sequence (question in QA) and y, is label (answer in QA). Additionally, there are LLMs accessible through black-box APIs [5, 8, 44, 43, 42]. In other words, the parameters and the architecture of the LLM are unknown and we can only access text sequences generated by the L.M.M Since the ability to generate high-quality rationale is known as the emergent ability of LLMs [55, 28], want transfer such ability to a small language model with reasoning distillation. Firstly, we leverage the chain-of-thought prompting [28] to elicit the proper I rationales for each training data point with LLMs: Tty- = LpM æ;, yi) for all E [n] == {1, (1,...,1) and E [I] where r is the generated rationale and is the chain-of-thouge prompt [55, 28, 48].
Fine-tuning Small Models on Rationales Then we fine-tune small language model pe with trainable parameters 0 generate both rationale Pış obtained from the LLM and answer yi, given the question x; In other words, we minimize the negative log- likeliooddood of the sequence of rationale Tsy and the answer y where the rationale must be generated first prior to the answer generation:
1
n
Laistill (0)
log po (Taj. ya x.)).
(1)
n

---

Training
Retriever
Trainable
Reranker
Trainable
Knowledge Dis
0.00
fp
Frozen
?
P6
External KB
0.32
B.72
Reranker
Rationale
Small LM
Candidate Set Dij & Score Q(dlriy)
Train Dataset
Large
Rationale
Question
Inference
fo Reranker
Retriever
Knowledge
Rationale
Extemal
Candidate Set D,
A
Small LM
Answer
Question
Question
Knowlodge
Figure 2: Overview of KARD. (Left, 4.1) Illustration of training (top) and inference (hottom) of knowledge- augmented reasoning distillation, where, during training. the small LM learns generate rationales given the training data and retrieved knowledge by the rationale. (Right, $ 4.2) Illustration of reranker training (top) and inference (bottom). Reranker learns to prioritize the passage which has knowledge relevant to the rationale.
Intuitively, the rationale provides deeper and more comprehensive understanding of the reasoning behind the answer associated with the question, which better guides the small model to correctly answer the question [18]. Although previous works [33, 17, 38, 12, 18] have also leveraged the ratio- nales generated by LLMS to make small models excel at diverse reasoning tasks, generating rationales for enowledge-intensii Section 3, the reasoning distillation with a small model but without knowledge augmentation may degrade the quality of the rationale generation due to the incapability of memorizing training data with the small model [57, 27] and the necessity of the memorization for better performance in language tasks [4]. Therefore, the rationale generation should be evidenced by extrinsic knowledge from external memory to enhance the capability of the small LM for generating high-quality rationale.
tasks with a small LM requires additional care. As previously described in
Integrating External Knowledge Base Motivated by Theorem we propose to retrieve a passage from an external Knowledge Base (KB) which is corpus of over millions of documents D {di,..., [ support memorization capacity of the small LM. Note that the acquisition of the relevant document from KB is crucial for training the small LM to generate high-quality rationale which leads correct answers for given questions. As done in open-domain QA task [7], we retrieve set of relevant passages for given query with the sparse retriever BM25 [46]. In order to obtain the document which is the most relevant the rationale rij generated by the LLM,, we utilize the rationale as a query to retrieve a set of passages D. = (opk(p(()rr)) D) k) C D, where p denotes retriever scoring the document d D based on relevance to the query Tij and topk yields the passages with the top-k highest relevance scores. Finally, we utilize the retrieved documents Diy for fine-tuning the small LM to generate the rationale Taj and answer y, for the question æ, follows:
Laistall-ga n log Po (rig. Uil 2; Diy),
where the rationale and answer are sequentially generated as we did Equation
4.2 Training Neural Reranker for Rationale Generation
The remaining issue that we cannot use the rationale as a query at the inference time. an alternative, we can use the question instead of the rationale rıj as a query to retrieve set of passages with the retriever. However, there is no guarantce that the top-k passages retrieved by the input æ, query contain relevant information generate correct rationales. In detail, based the question as a query, the retriever can obtain a set of passages that contain relevant documents for generating rationales with sufficiently large À but & < K. However, the target documents want for rationale generation may be assigned with low rankings and thus they may not be chosen for knowledge augmentation at the inference time. To remedy this issue, propose to leverage neural reranker fa [26] with parameter Ø to re-rank the set of passages retrieved by the retriever p so that can acquire more relevant documents for generating rationale at the inference time.
In order to train the neural reranker, we might manually construct a ground truth passage for each question. However, assume a realistic setting where the ground truth passage for reranker training
5

---

is not given. Instead, we train the reranker to imitate how the retriever scores the passage d E 'D with the rationale riy as query. Specifically, we first utilize the retriever p to obtain set of passages from "D with the rationale rij as a query as follows: Dij = topk p r:j D), 61) U ttopk(p(d] where к1 and K2 the number of candidate documents (Figure 2 is the case where K2 0). Then, normalize the score p(d]r.y; D) of the document from Dij, denoted as Q(d)r.)). Similarly, use the reranker fo score each document in s3 with the given question æ; and normalize the score denoted as Pa(d((a)) We use softmax for normalization with hyperparameters 71, 72,72 > 0 as follows:
D), 62)
Q(d(ri)) = Ea exp (p(d" raj: D)/T)² Pa(d(z) = sp(ff(d,xx)//1
exp (pid D)/T))
Laed, exp(f (d',,;))) x21lm²)
where d E Dig. Finally, we minimize the KL divergence between Q(d)r;y) and (dlax)):
1 n
Czeranx
Intuitively, the objective function guides the reranker to assign higher scores to passages that similar to the rationale raj. Note that both objective Lasserm-xa(0)) and (eeranz(o) are independent; therefore, do not need jointly update both of the small LM and the reranker.
4.3 Inference
After training. we obtain the small LM with the parameter 8" E argming Latatsii-ke(f and the reranker with the parameter d² argming Lrarante (o)) At the test time, answer the question w.. we first get a set of candidate documents D. = topk(p(d]x.; D), x*) with the retriever p and x² = 100. Then we re-rank all the document E D, with før and choose top-k relevant documents w.r.t the question æ. as follows: D. = topk((fo (d, | D.},A. k)). Finally, we generate a rationale r. argmax, Po (r x., D.) and an answer y. argmax, po- (y/r.,, D..D.).
5
Experiments
5.1 Experimental Setting
Task and Dataset In our experiments, we focus on wewllege-intensiigg reasoning tasks which require both the reasoning ability over the knowledge and the compound knowledge of the specific domain. As our primary benchmark, we use the medical multiple-choice question dataset MedQA- USMLE [23]. The dataset contains 12,723 4-option multiple-choice question answering problems from US medical licensing exam. This dataset is the best fit to evaluate our method since 98% of the questions simulate the realistic clinical settings by presenting patient cases that require extensive professional domain-specific knowledge and complex reasoning ability over multiple evidence sources. To further validate our approach, we employ Strategy QA [14] dataset, which involves 2,780 yes/no questions that demand sophisticated multi-step reasoning skills and the ability to gather supporting evidence from various domains. We additionally validate our approach on commonsense reasoning with Openbook QA [39] dataset, which consists of 5,957 elementary-level science questions with 4 multiple-choice options.
Baselines We compare our method against relevant baselines. Few-shot In-context Learning (ICL) utilizes few training samples as a prompt to make a prediction [5]. Few-shot ICL. + Chain- of-Thought (CoTT) leverages thain-of-thoughh prompting to generate rationale and generate an answer based on the rationale [28]. Fine-tuning refers to the one that fine-tunes a pre-trained model to generate answer given only a question. The performance of the above basclines represents the capability of small language model to solve knowledge-intensiee reasoning tasks using only training data but without any extrinsic guidance on reasoning external knowledge.
To assess the impact of external knowledge, we augment the above three baselines with documents retrieved from the knowledge base (Wikipedia), denoted as Knowledge-Augmented models. For the knowledge augmentation, we append retrieved passages along with the question at both training and inference time. These baselines help us understand how much external knowledge can improve the performance of each baseline. We also compare our KARD against the standard Reasoning Distillation without mmowledge-augmentaiin [33, 17, 38, 12, 18].

---

Table 1:
Experimental results on the MedQA-USMLE dataset with Flan-T5 [53] and OPT [20, 59] models. We
report the mean and standard deviation of accuracy with 3 different runs for reasoning distillation methods.
MedQA-USMMLE (Flan T5 [53])
MedQA-USMILE (OPT [20, 59))
Method
Base (250M)
Large (780M)
XL (3B)
350M
1.3B-IML
Few-shot
23.49
31.50
35.66
27.42
29.14
Few-shot + Chain-of-Thoueee (CoT)
25.22
32.21
32.99
25.06
26.39
Knowledge-Angmented Few-shot+ + CoT
31.34
32.60
34.41
25.84
28.75
Fine-tuning
30.71
34,49
37.39
26.47
25.77
Knowledge sogoledg--gggmentee Fine-tuning
33.39
37.71
39.12
25.84
28.67
Reasoning Distillation
31.03
39.62
46.322..6
29 29.43
44.302.00
KARD (ours, BM25)
+4
41,87
47.27
30,79
KARD (ours, Reranker)
38.15
44,59
48.94
32.86:1.12
38.83
KARD (Silver Knowledge, Oracle)
40.30
49.80
$3.50
35.90
42.15
CoT from ChatGPT (Teacher, Oracle)
61.59
65.51
67.16
50.27
Table 2: Experimental results on the StrategyQA and OpenbookQa QA dataset with T5 models [45]. indicates experiments with Flan-T5 having the same size. We report experimental results as in Table 1.
KARD
StrategvOA (TS (45])
OpenbookOA (T5 (45))
35
Method
Base
Large
XL
Base
Large
XL
Few-shol
48.47
48.47
51.67
23.00
27.60
25.00
Few-shot + Co'
48.47
48.33
48.76
27.60
27.40
27.80
KA Few-shol + CoT
48.47
48.91
48.76
27.60
27.60
27.80
Fine-funing
52.26
56.33
31.55
51.00
62.00
74.60
KA Fine-luning
52.11
58.81
53.38
53.80
64.60
73,80
25
Original
Pubmed
MedWiki
Reasoning Distallation
35.361
64.97, as
66
58.87 50 66. 66.15
77.00g.
Figure 3: Experimental results on
KARD (ours, Revonkor)
KARD (surs, RM25)
56.57 20 66.04. m 20.55
55,90+
65.944.1
68.8.1.0
29.33
55.05=
66.40....
64
401.71
78.53
76.00...2
MedQA-USM
where we per- form DAPT [15] on each dataset
KARD (Silver Kn. Orwele)
$7.50
65.65
72.34
63.10
72.40
82.00
CoT from ChatGPT (Oracle)'
66.38
67.10
72.05
58.60 60
78.80
87.80
of x-axis with Flan-T5 Base.
As oracle models, we present a variant of KARD that receives better knowledge as input. In particular, at the inference time, we augment KARD with the silver document which is the passage retrieved with the gold rationale generated by the L.LM as query. This model represents an upper bound of the neural reranker performance. Additionally, we directly provide the small instruction fine-tuned language models (Flan-T5 [53] and OPT-IML (20]) with the rationale from the LLM in inference, to assess the upper bound of the performance gain on small models with high-quality rationales.
Language Models For all the experiments, we use the T5 models [45] including Flan-T5 [9], and OPT models [59] including OPT-IML [20]. For the reranker, use LinkBERT models [56]. As for the teacher LLM, we employ GPT-3. 5-turbo (ChatGPT) [42] through the proprietary API.
See Appendix B for experimental settings in detail.
5.2 Experimental Results
Table I shows that KARD consistently outperforms all the baselines on the MedQA-USMLE dataset on both encoder-decoder (Flan-T5) and decoder-only (OPT) language models. Remarkably, KARD exhibits a substantial positive effect on smaller models, as evident from the significant performance gain of the Flan-T5 Base model, which has 250 million parameters, over fine-tuning baseline on the MedQA -USMLE dataset. Regarding the analysis of the model size, please refer to Section 5.3. The impact of KARD decreases as the size of the model increases since larger models can better memorize knowledge during pre-training and fine-tuning. Moreover, we empirically show that knowledge augmentation consistently improves performance not only in reasoning distillation but also in few- shot chain-of-the of-thughtt and fine-tuning. It is worth noting that this empirical evidence supports our theoretical analysis in Section 3 that knowledge augmentation enhances the performance of small models. Furthermore, our experimental results indicate that the reranker consistently improves the performance of models for all sizes, over the retrieval with BM25, From the experimental results with silver knowledge (oracle), there is room for improvement by retrieving more relevant documents, which can help the model generate high-quality rationale.
We also present additional experimental results on Strategy QA and Openbook QA datasets in Table 2. Once again, KARD outperforms all baselines in experiments with both datasets. Notably, compared to MedQA-USMLF, the few-shot methods on StrategyQA and OpenbookQA exhibit performance similar to random guessing, as T5 lacks the ability of in-context learning [9]. Furthermore, fine-tuning
7

---

Fine-tuning
KARD (ours)
In-vontext leurning (1IB)
42.5
60.0
47.5
50-
BM25
Reranker
Test ccunccy
G 45.0
[ 42.5
40
32.5
40.0
30-
$0.0
35.0
27.5
32.5
20
250M 780M
3B
Hoagg Hiad33 Hrags Hit-10
(a) Ratio of Training Data
(b) Model Size
(c) Hits@ metrics
Figure 4: (a) Efficiency on training data and (b) model size. On MedQA-USMI.R we compare KARD against
the fine-tuning bascline by varying either the number of training data with Flan-T5 Large or the number of parameters, including the few-shot in-context learning performance of Flan-T5 XXL (11B). (c) Considering silver documents as ground truth, measure Hits @k on the documents retrieved by BM25 and the reranker.
Table 3: Analysis rationale diversity.
Table 4: Analysis on...
Table 5: Analysis k.
BM25
Reranker
Flan-T5 Base
Rationales
Base
Large
Passages
Basc
Large
Passages
BM25
Reranker
=
30.09
35.43
x² 20
36.45
43.91
32.91
36.76
32.13
39.04
50
36.06
44.23
32.84
37.71
1= 10
32.91
41.79
R= 100
36.76
45.25
3
32.36
37.39
T5-XL on StrategyQA results in poor performance since it fails to generalize to the test data. On the other hand, reasoning distillation improves the performance of models across all different sizes both datasets. Our KARD further yields performance improvement over the reasoning distillation baseline, demonstrating the effectiveness of knowledge augementation in both datasets.
5.3 Analysis
Experiments with DAPT Domain-Adaptive Pre-Traine Tre-Training (DAPT) [15] is the useful strategy to adapt Pre-trained Language Models (PLMs) on the specific domain to effectively tackle the tasks on it, which is done by further pre-training the PLM on large-scale domain corpus 30, 37]. As is interesting to observe whether the DAPT can enhance the capacity of PLMs for reasoning distillation in domain-specife on relevant domain-specific data before distillation, we conduct experiments with models from DAPT. Specifically, we further pre-train the Flan-T5 Base model on two moderate-scale biomedical corpora, Pubmed abstracts and MedWiki [35], respectively. Then, we apply reasoning distillation and KARD PLMs with further pre-trained parameters. In Figure 3, we observe that DAPT Pubmed marginally enhances the performance of reasoning distillation. On the other hand, KARD contributes more substantially to performance improvement than DAPT. This result indicates that KARD offers distinct advantage in knowledge-intensive reasoning tasks compared to DAPT.
text
mnowledgeiintensing tasks by further performing training
Efficiency on Dataset and Model Sizes To validate the efficiency of our KARD terms of training data and model size, we measure the test accuracy on the MedQA-USMLE dataset while varying the number of training data and model parameters. As shown in Figure 4a, our KARD can effectively transfer the reasoning ability of the LLM with the proposed KARD mechanism, using only a small number of training data. Moreover, the gaps between the naive fine-tuning and our KARD become much larger when increasing the number of training data, which confirms that we can potentially increase the effectiveness of KARD with more training data for wowweedge-uugmenet from LLMss. Furthermore, it is worth noting that KARD is cmmple-efficient With 25% of the training data, KARD outperforms the same model fine-tuned on the full data.
distillation
For the efficiency in terms of the model size, as shown in Figure 4b, KARD with 250M parameters achieves higher accuracy than the fine-tuned model with 3B parameters (14 times larger). Moreover, KARD with 780M parameters outperforms the 11B in-context learning baseline. These results show the significant practical advantage of our KARD resource- restrrcereetticteed settings since the small LM with KARD requires significantly less computational cost yet it outperforms the LLM.
Retrieval Performance evaluate the performance of the reranker on MedQA-USMLE. we consider the top-3 silver documents retrieved with the rationales generated by LLM as the ground
8

---

Table 6: Rationale generation example of KARD, compared with Reasoning Distillation in MedQA-USMLE with Flan-TS XL. The answer blue. We mark the hallucination the generated rationale as yellow and highlight the factually correct statements in the generated rationale and its supporting fact in the passage as red.
clinic after several nonths of weight Sbe noticed 6.8 kg (15 uninicutional weight loss over the preceding several months. She has not changed her diet or exercise habits. She also reports feuding with her boyffiend over the temperature of their shared apartment, she always foels warmer than he does, The vital signs include: beart rate 110mmin and blood pressure 146/78 mm The physical exam notable for warm slightly moist Which the following is best single treatment option patient? (A) Glacceorienids (8) Methimazele (C) Programolol (D) Radioiadine therapy
Question: 29-ycca-old woman presents
Retrieved Passage
Outputs
Reasoning Distillation (without knoweledge)
No knowledge
The patient's is symptoms are consistent with hypenthy roidism, specifically Graves' disease. Glacocarticuids lre the firs--line treatment for Graves' disease Methima- zole, programolol, and radioiodine therapy can be used adjunctive treatment, Anewer: A
KARD (Ours, renanker)
Hyperthrmidism.. (..) The hast single treatment option for Graves' discuse is methimuzog, which may sempocurily help people while other trearments are having effect. Glacocorti- coids radioiodine therapy can also be used, hut they are not the best option in this case. Propmnnolal can help control the beart rate and blood pressure. but will not treat (..))
The symptoms and physical exam findings saggest hy- perthyroidism, specifically Graves' disease.
single trearment optioon for Grives" disense methima- zole, which works by inhibiting the thyroid hormone synthesis. Giucncortionids and induiiodine therapy can also used, hut they are not best option. Answer:
truth, and measure Hits@k k on the documents retrieved by BM25 and reranker with x* 100. In Figure 4c, the reranker achieves significantly better Hits @k k than BM25. This result indicates that the reranker successfully learns to prioritize passages that are helpful to generate correct rationale at the test time, which leads to performance improvement on the knowledge-intensive reasoning tasks.
The Number of Rationales During Training Following Ho et al. [17], we generate multiple rationales for each training sample order to facilitate diverse reasoning in small language model training. In Table 3, we present the impact of rationale diversity during training on both Flan-T5 base and large models using the MedQA-USMI. dataset. As the number of rationales per training data increases, the performance also improves, demonstrating the benefit of employing multiple rationales. However, the performance gains become small when we increase the number of rationals from 5 to 10. This suggests that utilizing more diverse rationales beyond 10 may not yield significant further improvements, at least in the MedQA-USMIL dataset.
The Number of Candidate Documents for Reranker It is crucial to determine the size of the candidate document set (K") to which the reranker assigns the relevance scores w.r.t a question. In Table 4, we present the performance of both Flan-T5 base and large models on MedQA-USMLE. while varying k². The results indicate that increasing the number of candidate documents tends to be beneficial, as it allows the reranker to consider a broader range of diverse candidate documents.
The Number of Passages Used for Inference Even LLMs tend to be easily distracted by irrelevant context [47]. Therefore. simply adding more passages during inference may not necessarily enhance performance if relevant knowledge is not selected. In Table 5, we present the impact of the number of passages used in KARD during inference (k in Section 4.3) on Flan-T5 Base and MedQA-USMA USMLE. We observe that the performance of KARD (BM25) without the re-ranker decreases with increasing k. This result implies that using additional passages does not always result in generating better rationales. In contrast, using two passages (k- 2) with the reranker is better than a single passage (k- 1). This result indicates that the reranker effectively selects more suitable knowledge than BM25, thereby contributing to performance improvement in the MedQA-USMLE benchmark.
Qualitative Analysis In Table 6, we provide an example comparing the rationale generated by our KARD against the rationale by the baseline model with reasoning distillation but without knowledge augmentation. We choose one sample from the MedQA- using the Flan-T5 XL model. The model without knowledge augmentation generates the rationale that seems plausible based on the given instruction and question. However, it mistakenly generates the hallucination that Glucocorticoids are the first-line treatment for Graves' disease, which is incorrect. As a result, it fails to predict the correct answer (B) Methimazole. In contrast, when the model is fine-tuned with KARD. it generates a correct rationale that is supported by the retrieved knowledge indicating that methimazole is the best single treatment option for Graves' disease. Consequently, it successfully predicts the correct answer. This example highlights the effectiveness of our KARD method for generating accurate rationales by incorporating relevant knowledge, which leads to an improved question answering performance on knowledge-intensie reasoning benchmarks.
dataset and generate the rationale

---

Discussion
6.1 Comparison to Retrieval-augmented Generation
Retrieval-Augmenntr Generation (RAG) [32] primarily focuses on solving knowledge-intensive tasks (c.g., open-domain QA), where the accurate knowledge retrieval is important to achieve higher performance. In terms of the methodology, the key dif- ferences between KARD and RAG are that RAG utilizes the question as a query and jointly fine-tunes the generator and re- triever. To quantitatively analyze the advantage of our KARD against RAG in reasoning distillation, we conduct experiments with RAG on the reasoning distillation with two datasets that we used in the main experiment, where we use Flan-T5 base for MedQA- USMLE and T5 base for Strategy QA as base LMs and DPR [25] as the trainable retriever for RAG. In Table 7, experimental results show that using RAG reasoning distillation achieves lower accuracy than KARD, showing that our KARD is more tailored approach to reasoning distillations.
Table 7: Experimental results includ- ing RAG on Reasoning Distillation (RD) with (Flan-)T3s basc.
MedQA StrategyQA
KA Fine tine-tnnngg
33.39
52.11
RAG + RD KARD (Renanker)
24.84
54.24
38.15
56.57
6.2 Failure Case Analysis
In Table 1, we can see significant differences between KARD with reranker on the Flan-T5 XL and the ChatGPT in MedQA-USMLE Our investigation focuses on understanding the cause these gaps by examining samples where our method fails while ChatGPT succeeds. We collect 30 samples from corresponding cases and categorize them into two groups The first group consists of cases where the reranker fails to obtain the document relevant to generating the correct rationale. The second group includes cases where the small language model fails to produce correct rationales and makes incorrect predictions, despite having access to relevant knowledge in the retrieved document. Out of 30 samples, 15 fall into the first category, while the remaining 15 belong to the second category. This observation indicates the need for further improvements in both retriever and distillation methods to enhance the performance of small language models in wwwwledge-intensiie reasoning tasks.
6.3 Limitations
We have shown substantial improvements in small LMs' performance on knowledge-intensive reasoning tasks through our KARD. However, it is important acknowledge the limitations of our study. First, in terms of methodology, the effectiveness of our knowledge augmentation heavily relies on the quality of the document retrieved from the external knowledge base. As indicated in Table and Figure 4c, our reranker substantially improves the performance of small models by retrieving better knowledge. Despite the diminishing performance gap between B.225 and the reranker as the model size increases, there is still significant difference between the document retrieved by the reranker and the silver knowledge. This indicates that the reranker might miss important passages that can augment the knowledge of even large LMs. Therefore, further ad advancements in retrieval methods are necessary to generate better rationale, as this remains an important research challenge even for large language models [13]. Second, regarding experiments, we have tested our approach on relatively small LMs having under 3B parameters given our limited computational budgets. However, exploring the use of relatively larger language models like GPT-3 [5, 44] or LLaMA [50, 51] with KARD could be of great interests, which is promising direction for future research.
7 Conclusion
In this work, we proposed Knowledge-Augmented Reasoning Distillation (KARD) which enhances the capabilities of small Language Models (LMs) on knowledge-intensive reasoning tasks that demand both knowledge and reasoning abilities. Our approach involves generating rationales from large LMs and fine-tunie tt--innng small LMs on these rationales, while augmenting small LMs with external knowledge from a non non-parametric memory. Our theoretical analysis motivates our method by demonstrating the effectiveness of external memory reducing the memorization requirements of small LMs. Through empirical experiments, we showed that KARD outperforms traditional approaches such as fine-tuning and reasoning distillation, thereby providing a pathway to improve small LMs in reasoning tasks that require a comprehensive understanding of domain-specifit knowledge.
10

---

Acknowledgements
This work was done while the first author was working at AITRICS. We would like to thank Kangwook Lee and the anonymous reviewers for their insightful comments and suggestions regarding this work, which helped us to make improvements to the paper. This work was supported by AITRICS, the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST). No. AS-2022-001187386 Development of Large Korean Language Model Technology for Efficient Pre-training, and No. No.2022-0-0071 the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. R8-2023-00256259 and NRF-2008RRAAAA1059922
and Samsung Electronics 02201224-08145-00)
References
[1] Parishad BehnamGhader, Santiago Miret, and Siva Reddy. Can retriever-augmeentd language models reason? the blame game between the retriever and the language model. arXiv preprint arXiv:2212.09144 2022. htt. org/ 10. 48550/arXiv 2212.09145
[2] Elliot Bolton, David Hall, Michihiro Yasunaga, Tony Lee, Chris Manning, and Perey Liang. Biomedlm. https: / /rrf..rfmm// stanford. edu/2022/ 1t/h//22/15////// http1///ppubbppppp htmll.
[3] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring. Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock. Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, ICML, pages 2240, 2022.
[4] Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When memo- rization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd annual ACM SIGACT symposium on theory of computing, pages 123-132, 2021.
[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[6] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference Learning Representations, 2023.
[7] Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. In Regina Barzilay and Min- Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Pupers, pages 1870-1879. Association for Computational Linguisties, 2017. URL https:t/ //oii wrg/oogg/110 /0. 18653/v1/P17-1177
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani

---

Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sunkaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaI.M: Scaling language modeling with pathways. arXiv preprint (...i:2224.0231111 2022. URL https: http:///oooor////////// 10. 18550/arXiv. 2204 02311.
[9] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling cnttrructiooffinetunn URL https://doi. org/1 10. 48550/ aarivv.r.vvv...////i 2210 11416.
language models. arXiv preprint arXiv:22 14416, 2022.
[10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christo- pher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1rXiv:2110.14168, 2021. URL https: /ttp///axvvvv/////////////aaaav/ www.aawwwwa.aaaaaaeeeeeeerweeeee wwg/abs/////ee/oogoo....o.e. /2100. 14168.
[11] Elias Frantar, Salch Ashkboos, Torsten Hoetler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL httpp::/oppereviiiiie
net f www.ooodoooorrrroooooo wd=tccppnfwss.......mm
[12] Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. arXiv preprint urXiv:2301.127200 2023. URL https: //doi.org/// 000.4445500///
2...2.21..227 12726.
[13] Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. Rarr: Researching and revising what language models say, using language models, 2022.
[14) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Trans. Assoc. Comput. Linguistics, 9:346-361, 2021. URL https: h/ap:://doo /doii org/10. 1162/
tacl_a_00377
[15] Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't t stop pretraining: Adapt language models domains and tasks. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 8342-8360. Association for Computational Linguistics, 2020. URL attps://don org/10. 11883/////222/2 acl-main. 7400.
[16] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wer Wei Chang. Retrieval augmented language model pre-training In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR, 2020.
[17] Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852-14882, Toronto, Canada, July 2023. Association for Computational Linguistics.
[18] Cheng-Yu Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8003-8017, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/v12202.. http://2222..innnn-eeeeee aindinggs-acl.507. URL https: http:////caaaaahaocaeco
00232023.00 findings-acl. ac1.507.
[19] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL. htttts:///pprrevies
12

---

[20] Srinivasan Iyer, Xi Victoria Lin. Ramakanth Pasunuru, Todor Mihaylov. Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O' O'Horo, Gabriel Pereyra, Jeff Wang. Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. OPT-IMI: scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.120111 2022. URL https://doo org/ 10. 48550/ arriv. 2212. 12017.
[21] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Ar- mand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https:// http://ppnneeieeeet///eoeoooeeee
[22] Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251): 1-43, 2023. URL http: //jmmr//r.r///// ooggg.rrrro.ooo.lo..oo..oo.lolll orgooog///oo
www.wowwr..//e..eeet..tt.t. pttpps/psssvv//ee wttper/s//2///3///00 0037 html.
[23] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang. and Peter Szolovits. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009. / 1081,, 2020. URL https://arxil: org/abst /bs/ 2009. 13081.
[24] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:200108861, 2020.
[25] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769-6781. Association for Computational Linguistics, 2020.
[26] Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextu- alized late interaction over BERT. In Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps. Vanessa Murdock, Ji-Rong Wen, and Yigun Liu, editors, Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 39-48. ACM, 2020.
[27] Junghwan Kim, Michelle Kim, and Barzan Mozafari. Provable memorization capacity of transformers. In The Eleventh International Conference on Learning Representations, 2023.
[28] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199-2277 2022.
[29] Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille ELepaño. Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, and Victor Tseng. Performance of chatgpt usmle: Potential for ai-assisted medical education using large language models. medRxiv, 2022. doi: 10. 12. URL https: poaateen//early/22///22222
//www. www.medrxiv..om
12. 19. 22283643.
[30] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: pre-trained biomedical language representation model for biomedical text mining. Bioinform., 36(4):1234 1240, 2020. URL httpp:://do /ttp:::://oioor//000. 1093/bioinf wtt.maatcsbbtt2.......
[31] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter- prompt tuning. In Marie-Francing Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3045-3059. Association for Computational Linguistics, 2021. URL https: //doi. org 10. 1ww11/11//v// v1/2021. canlp-main.220
13

---

[32] Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Rerrieval-augmented generation for knowledge-intenne intensive NL.P tasks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[33] SHIYANG LI, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jingu Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan. Explanations from large language models make small reasoners better. ArXiv, 0tss/2110006720 2022.
[34] Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther, Can large language models reason about medical questions? arXiv preprint arXiv:2207.08811 2022. doi: 10.48550/arXiv. 2207.08 00844. URL attpp://doo org/10 48550/8555K55 2207 01200.08800 08143.
[35] Valentin Liévin, Andreas Geert Motzfeldt, Ida Riis Jensen, and Ole Winther. Variational open-domain question answering. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 20950-20977. PMIR, 2023. URL
https: http:p::///ooceiing....m mlr Trees//v022/ievinng html.
[36] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference Learning Representations, ICLR 20/9, New Orleans, LA, USA, May 6-9, 2019, 2019. URL htthttts: httpap::///ppeeeviieeeee wtt.://www..........oooo.ooooo..
[37] Renqian Luo, Liai Sun, Yingce Xia, Tao Qin. Sheng Zhang, Hoifung Poon, and Tie- Yan Liu. BioGPT: generative pre-trained transformer for biomedical text generation and mining, Briefings in Bioinformatics, bbbc)4bb, 2022.
[38] Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 1773-1781, Toronto, Canada, July 2023. Association for Computational Linguistics.
[39] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and ichi Tsujii, editors, Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 2018, pages 2381-239) Association for Computational Linguisties, 2018. URL attps://dok: org/ 10. 18653/665vv http///v///1///// 1260.
[40] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M. Krumholz, Jure Leskovec, Eric J. Topol, and Pranav Rajpurkar. Foundation models for generalist medi- cal artificial intelligence. Nature, 616:259 265, 2023. URL httpp::/ddo http:////...ooo...../ 1038/ 441586-023-05882
[41] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz, Capa- bilities of GPT-4 on medical challenge problems. arXiv preprint arXiv:2303.1333 2023.
[42] OpenAl. Introducing chatgpt.
2022.
[43] OpenAl. GPT-4 technical report. arXiv preprint arXiv:2303 2023. URL https: oodoi.orrg/00 488500//500.
2303. 087774.
[44] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02 2022. URL. https://ddo http///ii.orgg////// org/10..1000 48550/ arXiv. 2203, 02155.
14

---

[45] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140: 1-140:67, 2020.
[46] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3(4):333-339, 2009.
[47] Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be casily distracted by irrelevant context. In International Conference on Machine Learning, pages 31210-31227 PMLR, 2023.
[48] Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Schärli, Aakanksha Chowdhery, Philip Andrew Mansfield, Blaise Agüera y Arcas, Dale R. Webster, Gregory Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13111 2022. URL 0t///100.855000 48550/arXiv. 2212.
[49] Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-38290, 2022.
[50] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Ro- driguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397) 2023. URL https: //doi.org///// 400.0/48550/// arXiv 2302 13971.
[51] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernan- des, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Sub- ramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams. Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint MrXXix233770098850 2023. URL https://dol org/10 10. 48550/arXiv /arXiv. 2307. 09288.
[52] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
[53] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: htttt::e///ppeeeeiieee
[54] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. Survey Certification.
[55] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
15

---

[56] Michihiro Yasunaga, Jure Leskovec, and Percy Liang. Linkbert: Pretraining language models with document links. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 8003-8016. Association for Computational Linguistics, 2022. URL https: /doi. org/10 10. 18653/v1/ 551.
[57] Chulhee Yun, Suvrit Sra, and Ali Jadbabaie, Small relu networks are powerful memorizers: a tight analysis of memorization capacity, Advances in Neural Information Processing Systems, 32, 2019.
[58] Jianyi Zhang, Aashiq Muhamed, Aditya Anantharaman, Guoyin Wang, Changyou Chen, Kai Zhong, Qingjun Cui, Yi Xu, Belinda Zeng. Trishul Chilimbi, and Yiran Chen. Reaugkd: Retrieval-augmented knowledge distillation for pre-trained language models. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2023, Toronto, Canada, July 9- 14, 2023, pages 1 1128 1 1136. Association for Computational Linguisties, 2023. URL https: /tw.///oi..o/// org/ 10. 18653/v1/2023
97.
[59] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: open pre-trained transformer language models. arXiv preprint arXiv:225.00066. 2022. URL https: /tt.///o././/. org/10gg1/00 48550/ 08550/arXia arxiv. 2205. 2205.01068.
16

---

Appendix
A Motivation: Effect of Knowledge-Augmenttton on Memorization
A.1 Additional Details and Discussion
We adopted the abstracted language problem. i... the next-symbol prediction problem with reference strings Cj ~ Uniform ({0, })"), considered in the main text of [4] with no symbol corruption. In this problem, the random process P ~ is defined by drawing Gj E {0, 1] uniformly at random as Oj ~ Uniform ((0, 1 }(d) for all j E [N]: a set {egbi=1 corresponds to a P. Then, the random process Y) ~ P is defined by sampling j ~ Uniform ([N]) and ~~ Uniform ({0, 2.......
. d 1}) and by setting Z (j. cy E)) and Y = (c 1), where ey 1) denotes the first symbols of ej and (1 is (1 + 1) symbol of o, The training data X ~ pen contains ((Z,, Y)) generated by the same process n times independently.
Recall that the inference algorithm uses a KB denoted by such that S = N R and ر S.S Here, S is
and not ordered; therefore do not know the identifier j of each E S and which 8 E S is useful given each (Z, Y). Thus, this still requires learning and memorizing some information from the training data X.
In Theorem 2, N (or R) is the number of useful (or unuseful) documents in KB to be extracted by retriever. Thus, N + R is the total number of possible documents to be retrieved. As N + R increases, we need to memorize more information the training data to get the best match at test time. Theorem 2shows that this increase happens only in the log rate, log2 (N R). Thus, we can get improvements when the size KB (d) is larger than the log of the number of possible choices to be retrieved by the retriever (NN R)).
Moreover, unlike Theorem 1, we can remove the dependence on n and have A(X))( O(N Tog, (N R)) in Theorem 2 when the training size n grows at faster rate than the number of the useful documents N. This is because our proof does not rely on the high probability of having a singleton training sample per each reference ej unlike the proof of [4].
A.2 Proof of Theorem 2
Proof. Let é > and m = logz((1 - ( ym) (N+R)2-(N+R)) ). Since the distribution over subpop- ulation is uniform,
2c
N Σ Pr(o)Q.)(E
N
err q.n
(A)
Σ Pr(EolQ,).
5=1
where Eg is the event of c(Z A(X),)) + and Q2 is the event that the subpopulation of the test sample (),Y) ~ P is j. Let Ey be the event of having at least one training data sample ((Z,, Y,) X) in the subpopulation j of the test sample (Z, Y) P. Then,
Pr(FolQ,) Pr( Pr(Fl1))) Pr (r(Fol FEj + Pr (F)))) Pr(Ea) FY)
where Fij is the complement of E.. Denote by e the length of training sample, and by E the length of the test sample, the subpopulation j. Let Ey be the event of & > for least one training sample the subpopulation j, and Fy' be the event of e. < &g for at least one training sample the subpopulation Then,
N
erro g.n (A)
P(EE)))) BY) Pr(Eo E3)
N
j=1
PLE ECC,,, E2)
N
17

---

Pr(B:IQ)) Bi).
Define Enj to be the event of Morr(( 2) # Y where MorT Aorr(X). Then, the same decompo- sition holds for errg, (AoPt) yielding that
N
err (.Tg,((APPr)
P
==1
N
N P(C F 3,8
N
Pr(E(()))Pr(E(()) B7).
N
Since the probabilities are non-negative, by ignoring some terms, we have that
1 PrEE(O)))((Elll,,..)
errg.n (AOPT) 2
N
N
+ N PrE3)) PT Ej E
Σ
EY,
Since these two terms Pr(Fij Qj. F.) and Pr(Fi Fi, Fl Fg)) correspond the errors when the test data comes into the place with no information from the training data, the memorizing data does not help and the random guess achieves the best performance. Moreover, these two cases can be detected at test time by (A) by memorizing the first min(m, P bits of an input sequence in the training data and (i.e., min (ii, k bits in total) for at most one training sample per j E [ Thus, we choose A to memorize the first min (m, Ø. bits and for at most one training sample per j [N]; i.e., we memorize min(m, (.) + 1 S m + bits for at most min(N, n) training samples. In total, we memorise most min(N, (in(Nnn))m mm + 1) bits. Then, these two terms the same for err e (AoPT) and errif
q.n (A) by detecting these cases (with memorization) and generating random guess for these cases. Thus,
N
on 2 PEEa)) N
j=1
1
P(ER)))(HEECCEEEHHHHHHAHAAAPAPP Ez).
N
Therefore,
erry (A) err (
N
P1 Qj E3)
1
18

---

N Σ
Pr(ul) B() j=1
Pr(E PrEE000)PREEE00,0)))(000,,,,,,,,
Pr 14
Eg, E3)
0 because A memorizes the first min(m, ( bits, where min (m, 2.) &g conditioned on E2 and because this is conditioned on the events Ej and Eg. That is, Pr(Eo) Qj Ey. Eg, E3) is the error probability when we have a training sample with the length &g larger than the test sample length E/ on the same subpopulation and & = min(m, f bits of the training samples are memorized. In this case, we can simply output the next symbol for the test sample from the memorized training sample, yielding Pr(Fo Qq Eq, Eg EE) = O. Therefore,
Here, notice that Pr Ei
N Σ ==1
erry (A) - - (. S
PrE Pr Pr((E))))))001000 E/. EJ).
(4)
N
In the case of the events Qj. Ej and Eq. A(X) memorized min(m, (2) = m bits of the input sequence in the training sample for at least one training sample in the subpopulation j of the test sample. In this case, at test time, & first chooses one training sample in the same subpopulation j of the test sample (we can do this because the non-label part Z contains the subpopulation information j in both test sample (2, Y) ~ P' and in the training sample; see Brown et al. 4). Then, io uses the memorized m bits of this training sample to search S; i.e., it picks s E S such that the first m bits of s matches with the memorized m bits of the training sample. If there are more than one E S that satisfies this condition, then it randomly picks one of them. Note that in this case of the events Qj, Ei and Ez, there is at least one match; there exists 8 E S the first m bits of 8 matches with the memorized m bits of the training sample.
Thus, if there is no more than one E S that satisfies this condition, then the inference algorithm y uniquely determines the reference string oj of the test sample (2, Y) ~ P. With the reference string Cy of the test sample, ip then predict the next symbol of the test sample by outputting (e) + 1)-th symbol of the reference string cj. where & is computable by computing the length of the test input Z without any information of training samples. Therefore, if there is no more than one E $ that satisfies the above condition, then yp makes correct prediction. This implies that s and s' do not have the same first m bits for all s E S with s a s', then makes a correct prediction. By taking complement, this means that the error occurs only if there exist 8, 5 E S with 8 + s such that s and s' have the same first mn bits. Therefore,
Pr(EolQ; E/. E3)= < Pr(3s, s' E S with a s' such that and s' have the same first m bits)
= P is and s' have the same first m bits})
Σ
Pr(ts and s' have the same first m bits})
[S
N +
R
1
(N + R) (N + R) 1
2
2
2m
where T = 5/ E S, + s'}. Since m = log, (11 N-1 )rn (N+R) (N-R)
2
(N + R)2 - (N + R)
1
Pr(Eo Qj, Ey. E3) <
2
(1 N-1 )r) (N+R)= 100-=-N+00 (N+8)
N-1 )m
19

---

Plugging this into equation 4,
N
err q.n (A) err, er (AOPT) - Pr(B()i))PP( E
1
Σ
c
N
(N
j=1
Since Pr(E3 Qj. F3) - and Pr(F((Q;) = Pr Pr(BilQ)) =
( (N-1)n
err (A) ert (AOoT)
Σ j=1
E c
N
Since we only require m + bits for at most min(N, n) training samples from the above construction of A and ip, we have that I(X;A(X))P) erry q,m (A) (AOo) S e for any e 0.
n()m while achieving
B Implementation Details
Rationale Generation As for the teacher LLM, we employ UPT-3.5-turbe. (ChatGPT) [42] through the public API. We demonstrate the prompt we used for rationale generation in Table 11, 12, and 13. Specifically, we utilize the instruction and 5-shot examples from Singhal et al. [48] for McdQA-USMLE, and the chain-of-thought prompt Kojima et al. [28] for StrategyQA and Open- bookQA. We generate multiple rationales for each training sample with the LLM. This allows training the small LM with diverse set of rationales. Furthermore, to remove incorrect rationales from the training set but we use a small Flan-T5 base to filter the rationales which lead to incorrect prediction.
utilize the filtering method [17]
Training For all our experiments, we fine-tune the small language model for 3 epochs with a batch size of 32 using the AdamW optimizer [36] and a learning rate of 10-4, Each model utilizes a maximum of 96GB GPU memory with NVIDIA TITAN RTX GPUs for fine-tuning. In the StrategyQA and OpenbookQA experiments, potential data contamination with the corresponding test set, as Flan-T5 is fine-tuned on both datasets during instruction tuming. For the number of documents used for knowledge augmentation during KARD training, we set specifically, append documents retrieved from the retriever p along with each training sample to construct the input for training. See Tables 14 and 15 for examples of the input and output used in KARD training. For the train-test split of dataset, and OpenbookQA [39]. For strategy QA, we split the training set into 7 : 3 ratio to build the in-house test following Ho al. [17].
use the T5 model instead of Flan-T5 15n prevent any
for MedQA-USMLE and StrategyQA and h = 3 for OpenbookQA-
the official split for MedQA-USMLE [23]
Inference When it comes to methods that require rationale generation, such as Chain-of-Thoughh and reasoning distillation, employ technique called self-consistency [52] during inference. Specifically, for each question, model generates multiple rationales and corresponding predictions during the inference, followed by majority voting choose the final answer among the predictions.
Retriever use Wikipedia as the external knowledge base for both all of datasets. For the retriever D. we use BM25 [46] which is the frem-ffqquenc--baee sparse retrieval method. To implement BM25, we use the pyserini library² which provides reprodueible information retrieval framework.
Reranker To implement a neural reranker, we adopt the scoring method used in ColBERT [26]. We use BioT-nkBERT-base and LinkBERT-base [56] as the backbone language model for MedQA- USMLE and StrategyQA, respectively. For reranker training, we utilize LoRA [19] for efficient training. For hyperparameters, we set 7 and 72 100. For all datasets, we train the reranker for 3 epochs with a batch size of 16 using AdamW optimizer and a learning rate of 10-4, We set N1 = 8, K2 for MedQA-USMLE and in 4, K2 for StrategyQA and OpenbookQA.
20

---

Table 8: Analysis retriever.
Table 9: Analysis on K1 and Ki.
Table 10: Analysis on queries.
Flan-T5 Base
Retriever
Wikipedia
Pubmed
K1
Flan-T5 Base
Flan-T5 Base
Query
BM25
Reranker
BM25
33.14
31.58
K1 =
Rg = 4
36.84
DPR
29.77
0
8
34,96
31.81
35.51
Reranker
38.15
36.84
K1
R2 0
37.71
rij
32.91
37.71
Silver (oracle)
40.30
45.48
DAPT In Section 5.3 Experiments with DAPT, we conduct domain-adaptive pre-training [15] to assess the impact of further pre-training prior to the reasoning distillation. We further pre-train Flan- Praetts
T5 base model the LM objective discussed in the original T5 and prompt-tuning paper [45, 31].
We use two different corpus for this experiment. One is the PubMed abstracts' which contains the 1.8 abstracts extracted from the biomedical papers uploaded in PubMed. Another is MedWiki which is the subset of the Wikipedia passages containing the biomedical knowledge [35]. We further train the model for epochs with batch size of 128 using the Adam optimizer and learning rate of 10
C More Analysis
C.1 Retriever and Knowledge Base
In our main experiments, we utilize BM25 [46] as the retriever and Wikipedia as the knowledge base. Regarding these choices, several questions arise: (1) Does using dense retriever provide more advantages than sparse retriever? (2) Does utilizing domain-specific knowledge base offer more benefits than general knowledge base?
To address these questions, we perform an analysis on the choice of retriever and knowledge base. First, we substitute the external knowledge base with Pubmed abstracts corpus", which consists of abstracts from publicly available medical papers containing specialized knowledge. On the other hand, we maintain the knowledge base Wikipedia but replace the BM25 retriever with DPR [25], one of the popular dense retrievers in open-domain question answering tasks.
In Table 8, we present the results of our analysis. We observe that the use of PubMed does not result in performance improvement compared to Wikipedia. We speculate that incorporating a reranker with PubMed corpus could potentially yield further improvements considering that using silver knowledge from PubMed yields superior performance against using silver knowledge from Wikipedia; however, leave it future work. Furthermore, using DPR rather leads degraded performance. We hypothesize that DPR may struggle to generalize to our specific task since is primarily designed for open-domain QA tasks. It is plausible that employing more adaptable retriever such as contriever [21] might offer greater benefits, but we also leave this as an avenue for future research.
C.2 K1 and K2 for Reranker Training
In the training of the reranker (as described in Section 4.2 of the main paper), the reranker Iearns to prioritize passages that are relevant to the rationale. We normalize the score p(d[r,y; of the passages in the candidate set through softmax. Therefore, the reranker fa learns a relative score fdl a.:) between passages in the candidate set during training. Due computational constraints, the number of passages in the candidate set is limited. As a result, it is necessary to determine which passages should be included in each candidate set Diy-
There are two potential sources for the candidate passages. One approach is to retrieve candidate passages using the rationale rıy as query, while another approach use the question (input data) 2; the query. is intuitive to consider combining candidate passages from both approaches, this can provide additional information to the reranker. Specifically, the reranker can learn the differences between passages that are highly similar to the question and passages that are highly similar to the
>nctpe://huggingface. http://acaac/aaaaaaa////cc/////a _bsttaact_ "nttps://pubmed" ncbi nlm, nih. gov/
21

---

rationale. This training setup enables the reranker to prioritize passages that are more relevant to the rationale among the passages retrieved from the question as a query during inference time.
In the implementation, we can control the composition of the candidate set by adjusting hyperpa- rameters Ki and K2 while fixing the candidate size as 8. In Table 9, we provide an analysis of the impact of both values on the reranker performance, by measuring the task accuracy with the F1an-T5 base model. We observe that relying solely on passages related the question for composing the candidate set is not a viable approach, as it hinders the reranker's 's ability to learn which passage is relevant to the rationale effectively.
C.3 Knowledge used in KARD training (Di))
In order to enhance the reasoning ability of a small LM, it is crucial to retrieve passages that contain the proper knowledge assist the small LM in generating rationales obtained from the large language model. It is intuitive to utilize the rationale itself as query for retrieval, focusing to retrieve passages that are relevant to the given rationale. To validate this intuition empirically, we instead use the question as a query for the retrieval during training, retrieving passages denoted as topk(p(d a.;; D), k:). In Table 10, we present the empirical evidence that using the passages relevant to the question during training actually leads to performance degradation, as these passages are highly unlikely to contain the necessary knowledge for generating rationales.
C.4 Examples of Failure Cases
In Section 6.1 of the main paper, we provide an analysis of the failure cases. In this section, we showcase an example for each category of failure cases.
The first category involves the case where the reranker fails retrieve pertinent knowledge required for generating a rationale. In Table 16, we present an example of the failure case that corresponds to the first category. In this example, the reranker fails to retrieve the relevant passage necessary for generating the correct rationale. Therefore, the small Language Model (LM) generates rationales that include a hallucination: suggesting warm compresses as the most suitable treatment for allergic conjunctivitis, which is factually incorrect.
The second category is the case where the reranker successfully retrieves the relevant passage which is the silver passage obtained from the gold rationale from ChatGPT. In the example illustrated in Table 17, the retrieved passage contains information about pulmonary contusion, which is one of the answer options. If the small LM comprehends the given passage accurately, it should recognize that pulmonary contusion is not the correct answer, as elevated pulmonary capillary wedge pressure and troponins are unrelated to pulmonary contusion but rather associated with a cardiac contusion. However, despite being trained with KARD, the small LM fails to reason based on this knowledge. resulting in the generation of the incorrect rationale.
Furthermore, we also provide examples of failure cases from StrategyQA. Unlike MedQA-USMLE, most of the failure cases in OA fall into the first category, where the reranker fails. This is due to the fact that StrategyQA questions often require complex and multiple sources of knowledge to answer.
In Table 18, we present an example that corresponds to the first category. The question requires knowledge about the background color of the Azerbaijani flag as well as the color composition of the Powerpuff Girls. However, the reranker retrieves passage related to the flag of Japan, which is not helpful for answering the given question.
In Table 19, we show the case that belongs to the second category, which is rare case in Strategy In this example, the reranker successfully retrieves a passage containing information about the number of fused vertebrae in human sacrums. However, the small LM fails to answer the question due to its inability to comprehend the retrieved passage. Moreover, since the question requires additional knowledge about the number of fused vertebrae in the sacrums of an Alaskan Malamute, is challenging to answer the question solely based on the available knowledge, particularly if the small LM lacks intrinsic knowledge regarding this specific domain.
22

---

D Broader Impact
Our proposed approach aims to enhance the performance of small language models in tasks that involve knowledge-intene intensive reasoning. As demonstrated in Section 5 of the main paper, our method beneficial in domains that require professional knowledge, such the medical field.
However, is crucial to exercise caution when employing small language models with our method in real-world clinical applications. These models have the potential to generate statements that factually incorrect, as explicitly mentioned in Section of the main paper and Section C.4. Therefore, thorough attention and careful consideration are required when utilizing small language models in such contexts, even with our proposed method.
It is worth noting that even large language models, which have not been extensively validated in real-world clinical settings, should also be used with caution. The potential for generating inaccurate information exists across various language models, and their deployment in clinical sites should be approached with careful consideration and validation [48, 41, 40).
23

---

Table 11: MedQA [23]. 5-shot Chain-of-Thought prompt [48] for rationale generation with ChatGPT [42].
The following are multiple-choice questions about medical knowledge. Generate detailed step-by-step explanation for cach question and answer.
Question: A 22- 22-year-old male marathon runner presents to the office with the complaint of right-sided rib pain when he runs long distances. Physical examination reveals normal heart and lung findings and an exhalation dysfunction at ribs 4-5 on the right. Which of the following muscles or muscle groups will be most useful in correcting this dysfunction utilizing direct method?
(A) anterior scalene (B) latissimus dorsi (C) pectoralis minor (D) quadratus lumborum
Answer: (C)
Explanation: We refer to Wikipedia articles on medicine for help. Among the options, only pectoralis minor muscle origins from the outer surfaces of the 3rd to 5th ribs.
Question: 36-ycar-old male presents the office with 3-week history of low back pain. He denies any recent trauma but says that he climbs and out his truck numerous times a day for his job. Examination of the patient in the prone position reveals deep sacral sulcus on the left, posterior inferior lateral angle on the right, and a lumbosacral junction that springs freely compression. The most likely diagnosis is (A) left-on leftett sacral torsion (B) left-on-right on-right sacral torsion (C) right unilateral sacral flexion (D) right- on on-right
sacral torsion
Answer: (D)
Explanation: We refer to Wikipedia articles on medicine for help. The deep sulcus on the left, a posterior
II.A on the right, with negative spring test suggests a right-on-right sacral torsion. All other options have a deep sulcus on the right.
Question: A ye-year- man comes to the office because of a 3-day history of sore throat, nonproductive cough, runny nose, and frontal headache. He says the headache is worse in the morning and ibuprofen does provide some relief. He has not had shortness of breath. Medical history is unremarkable. Hc takes no medications other than the ibuprofen for pain. Vital signs are temperature 37.4 37.4% (99.4'F), pulse 88/min, respirations 18/min, and blood pressure 120/84 mm Hg. Examination of the nares shows erythematous mucous membranes. Examination of the throat shows erythema and follicular lymphoid hyperplasia on the posterior oropharynx. There is palpable cervical adenopathy. Lungs are clear to auscultation. Which of the following is the most likely cause of this patient's 's symptoms?
(A) Allergic rhinitis (B) Epstein- Barr virus (C) Mycoplasma pneumonia (D) Rhinovirus
Answer: (D)
Explanation: We refer to Wikipedia articles on medicine for help. The symptoms. especially the headache, suggest that the most likely cause is Rhinovirus. Epstein- Barr virus will cause swollen lymph nodes but there is palpable cervical adenopathy. Lungs are clear auscultation suggests it's Mycoplasma pneumonia.
Question: A previously healthy 32-year-old woman comes to the physician 8 months after her husband was killed in a car crash. Since that time, she has had a decreased appetite and difficulty falling asleep. She states that she is often sad and cries frequently. She has been rechecking the door lock five times before leaving her house and has to count exactly five pieces of toilet paper before she uses She says that she has always been a perfectionist but these urges and rituals are new. Pharmacotherapy should be targeted to which of the following ncurotransmitter?
(A) Dopamine (B) Glutamate (C) Norepinephrine (D) Serotonin
Answer: (D)
Explanation: We refer to Wikipedia articles on medicine for help. The patient feels sad and among the options, only Dopamine and Serotonin can help increase positive emotions. Serotonin also affects digestion and metabolism, which can help the patient's 's decreased appetite and sleep difficulty.
Question: 42-year-old man comes to the office for preoperative evaluation prior to undergoing adrenalec- tomy scheduled 2 weeks. One month ago, he received care in the emergency department for pain over his right flank following motor vehicle collision. At that time, blood pressure was 160/100 mm Hg and CT scan of the abdomen showed an incidental 10-em left adrenal mass. Results of laboratory studies, including complete blood count, serum electrolyte concentrations, and liver function tests, were within the reference ranges. The patient otherwise had been healthy and had never been told that he had elevated blood pressure. He takes no medications. A follow-up visit in the office 2 weeks ago disclosed elevated urinary normetanephrine and metanephrine and plasma aldosterone concentrations. The patient was referred to surgeon, who recom- mended the adrenalectomy. Today, vital signs are temperature C (97.9*F), pulse 100/min, respirations 14/min, and blood pressure 170/95 mm Hg. Physical examination discloses no significant findings. Initial preoperative preparation should include treatment with which of the following? (A) Labetalol (B) A loading dose of potassium chloride (C) Nifedipine (D) Phenoxyhenzamine Answer: (D)
Explanation: We refer Wikipedia articles medicine for help. The symptoms and the adrenal mass suggested pheochromocytona. and the blood pressure indicates hypertension.
is used to
treat hypertension caused by pheochromocytoma.
Question: [question] Answer: [answer] Explanation:
24

---

Table 12: Strategy [14]. O-shot Chain-of Thouuhh prompt [28] for rationale generation with ChatGPT [42].
The following multiple choice questions (with answers). Generate detailed step-by-step explanation for each question and answer.
Question:
[question]
Answer: [answer]
Explanation: Let's think step by step.
Table 13: OpenbookQA [39]. 3-shot Chain-of-Thought prompt for rationale generation with ChatGPT [42].
The following are multiple choice questions (with answers). Generate detailed step-by-step explanations for each question and answer.
The sun is responsible for (A) puppies learning new tricks (B) children growing up and getting old (C) flowers wilting in (D) plants sprouting, blooming and wilting
Question:
Answer: (D)
Explanation: We refer to basic knowledge about the effects of sun on living organisms. The sun provides the energy required for photosynthesis in plants, which allows them sprout, bloom, and eventually wilt. The other options are related the effects of the sun.
Question:
When standing miles away from Mount Rushmore
(A) the mountains seem very close (B) the mountains are boring (C) the mountains look the same from up close (D) the mountains
smaller than in photographs
Answer: (D)
Explanation: This question requires some basic knowledge about perspective and the way our eyes perceive distance. When we stand miles away from an object like Mount Rushmore, it appcars smaller than it does photographs because eyes capturing smaller visual angle. Option (A) is incorrect because if the mountains seemed very close, we would perceptual phenomena. Option (C) untrue hecause when we are up close to Mount Rushmore, we can see details that we cannot see from far away.
standing miles away. Option (B) subjective and related
When food is reduced in the stomach (A) the mind needs time to digest (B) take a second to digest what (C) nutrients are being deconstructed (D) reader's digest a body of works
Question:
Answer: (C)
Explanation:This question requires basic knowledge about the digestive system. When food is in the stomach, it broken down and deconstructed into nutrients that absorbed by the body. Option (A) incorrect because mind is not directly involved the digestive process. Option (B) is play on words and not related to digestion. Option (D) is a reference to literary magazine and not related to the digestive process.
Question: [question] Answer: [answer] Explanation:
25

---

Table 14: Input and output example for KARD (training) in MedQA-USMLE [23].
INPUT:
The following are multiple-choice questions about medical knowledge. Generate a stop-by-step explanation for each question:
Question: A 23-ycar-old pregnant woman at 22 weeks gestation presents with burning upon urination. She states it started day ago and has been worsening despite drinking more water and taking cranberry extract. She otherwise feels well and is followed blood pressure is 122/77 mmHg. pulse 80/min, respirations are 19/min, and oxygen saturation is 98% on room air. Physical exam is notable for an absence of costovertebral angle tenderness and gravid uterus. Which of the following is the best treatment for this patient?
doctor for her pregnancy. Her temperature 97.7 (36.5*C) C),
A. Ampicillin B. Ceftriaxone C. Doxycycline D. Nitrofurantoin
Knowledge: Urinary tract infection. Urinary tract infections are concerning in pregnancy due the increased risk kidney infections. During pregnancy, high progesterone levels elevate the risk decreased muscle tone of the ureters and bladder, which leads to a greater likelihood of reflux, where urine flows back up the ureters and towards the kidneys. While pregnant women do not have an increased risk of asymptomatic bacteriuria, bacteriuria present they do have 25-40% risk of kidney infection. Thus if urine testing the absence of symptoms trrmptonnstteratmenttt recommended. Cephalexin nitrofurantoin are typically used because they are generally considered safe in pregnancy. A kidney infection during pregnancy may result in premature birth or pre-eclampsia (a state of high blood pressure and kidney dysfunction during pregnancy that can lead to seizures). Some women have UTIs that keep coming back in pregnancy and currently there is not enough research on how best treat these infections.
shows signs
infection-eve
Explanation:
OUTPUT:
We refer Wikipedia articles medicine for help. The patient is pregnant, which limits the options for treatment due to risk harm to fetus. Nitrofurantoin considered safe for use in pregnancy and an effective treatment for uncomplicated urinary tract infections. Ampicillin and Ceftriaxone are not typically used to treat UTIs, and Doxycycline is contraindicated in pregnancy.
Answer: D
Table 15: Input and output example for KARD (training) in StrategyQA [14].
INPUT:
The following are multiple-choice questions. Generate a step-by-step explanation for cach question:
Question: Is the cuisine of Hawaii suitable for a vegan?
yes B. no
Knowledge: Cuisine of Mariana Islands The cuisine the Mariana Islands is largely made up of meat dishes including pork, poultry, horse meat, and meat from other land mammals. Some popular land animals consumed include bats, mice, rats and lizards. Most of the dishes on the islands are cooked in coconut oil. Naturally, the islands' surrounding waters make seafood another popular option. Some seafoods include sea cucumbers, whale's meat, and various fish. said that Mariana Island's cuisine heavily influenced by its neighbors Papuan, Hawaiian and American cuisines. The Mariana's 's cuisine is very international as well. The people enjoy many Asian dishes such as kimchi. Explanation:
OUTPUT:
Hawaian cuisine typically includes a lot of seafood, pork, and other types of meat. These ingredients are not suitable for a vegan as they come from animals. While there may be some plant-based options cuisine like fruits, vegetables, and grains, the majority of dishes are not vegan-friendly Therefore, the correct answer is (B) no.
in Hawaiian
Answer:
26

---

Table 16: McSQA-USMME Example Failure Case (frrt category). We compare the generated output from the Flan-T5 XL model with KARD and reranker against the gold correct rationale from ChatGPT. The answer is in blue. We mark the hallucination in the generated rationale as yellow and highlight the factually correct statements the generated rationale and its supporting fact in the passage as red.
Question: A 35-year-old man comes to the physician because of itchy, watery eyes for the past week. He has also been sneezing multiple times day during this period. He had similar episede year ago around springtime. He has iron deficiency anemia and ankylosing spondylitis. Current medications include ferrous sulfate, artificial tear drops, and indomethacia. He works elementary school teacher. His vital signs are within normal limits. Visual acuity is 20/20 without correction. Physical examination shows bilateral conjunctival injection with watery discharge. The pupils are mm. equal, and reactive to light. Examination of the anterior chumber of the eye is unremarkable. Which of the following is the most appropriate treatment? (A) Erythromycin ointment (B) Ketotifen eye drops (C) Warm compresses (D) Fluorometholone eye drops
Retrieved Passage
Outputs
KARD
Mansonella perstans A 36-year-old man was admitted to the outpatient clinic the Goundi Missionary Hns- pital the south of Chad May 2001. He complained of visual impairment in the left eyc, ocular and abdom- previously
We refer Wikipedia articles medicine for help. The symptoms suggest allergic conjunctivi- tis, and the most appropriate treatment is warm compresses for comfort and relief, Frythromycin ointment is used for bacterial conjunctivitis. kete- tifen eye drops used glaucoma, and fluo- rometholone eye drops are used for dry eyes. An- swer:
(Ours, remnker)
inal pruritus, and abdominal pain. been treated with DEC for 'M. perstans" infection five months prior to his visit. A blood sample was taken 11:00 am, and examined microscopically as thick blood film stained with Giemsa'' 's solution. The thick blood film revealed the presence of 'M. perstans", and other parasites were found. He had 3% cosinophilia. A visual acuity test showed a reduction of visual acuity the left eye, while right eye was 9/10.
(..)
Gold (Oracle)
Emedastine... Emedastine (trade name Fimadine) is a second generation antihistamine used in eye drops to ulleviate the symptoms of allergic conjunctivitis It acts as receptor antagonist. works by blocking the action of histamine that causes allergic symptoms. is used in form of the difumarate. The emedastine difumarate white, erystalline, water-soluble fine powder. Emedastine eye drops is usually applied twice day the affected eye. When patients with aller- gie conjunctivitis were treated with 0.05% emedastine difummaraee ophthal solution for six weeks, the signs and symptoms such redness, itching and swelling of the eyes were relieved. Emedastine appears to be devoid effects on adrenergic, dopaminergic and sero- tonin receptors. This drug developed by Alcon, which is global medical company specializing in eye care products.
We refer to Wikipedin articles on medicine for help. The patient's 's symptoms suggest allergie con- junctivitis, which can treated with antihistamine eye drops like Ketotifen. Frythromccin ointment used to treut bacterial infections, warm compresses may help with symptoms but do not treat the un- derlying issue, and Fluorometholone eye drops used to treat inflammation and do not address the allergic response. Answer:
27

---

Table 17: MESQA-USMLE Example of Failure Case (second category). We compare the generated output from the Flan-T5 XL model with KARD and reranker against the gold correct rationale from ChatGPT. The answer is in blue. We mark the hallucination in the generated rationale yellow and highlight the factually correct statements in the generated rationale and its supporting fact in the passage as red.
Question: A 27-year-old man presents to the emergency department after motor vehicle collision. The patient was the front seut unrestrained driver in head on collision. The patient's 's echocardiogram (ECG) is notable only for sinus tachycardia. His temperature 99.5*F (37.5*C) blood pressure 107/58 mmHg. pulse 120/min. respirations are 17/min, and exygen saturation 98% on room air. The patient is given liters of Ringer lactate solution and morphine. Initial workup demonstrates that the patient's s pulmonary capillary wedge pressure and troponins are elevated. The patient is currenly complaining of chest pain. Physical exam is notable for an uncomfortahle young man with hruising over his chest wall. Which of the following the most likely diagnosis? (A) Cardiac contusion (B) Hemorrhage (C) Pulmonary contusion (D) Takotsubo cardiomyopatly
Retrieved Passage
Outputs
KARD
Pulmonary contusion... pulmonary contusion, also known lung contusion, bruise of the lung. caused by chest trauma. As a result of damage to capillaries, blood and other fluids accumulate The excess fluid interferes with gas exchange, poten- tially leading to inadequate oxygen levels (hypoxia). Unlike pulmonary laceration, another type of lung in- jury. pulmonary contusion does not involve a cut or of the lung tissue, pulmonary contusion usu- ally caused directly by blunt trauma but can also result from explosion injuries or a shock wave associated with penetrating trauma. With the of explosives during World Wars I and II. pulmonary contusion re- sulting from blasts gained recognition. occurrence in civilians began to receive wider recog- nition, in which cases it is usually caused by truffic accidents. The of the risk to vehicle occupants. Dingnosis is made by studying the cause of the injury. physical examination and chest radiography. Typical signs and symptoms include direct effects of the physical trauma, such chest pain and coughing up blood, as well as signs that the body is not receiving enough oxygen, such as eyanosis, The contusion frequently heals on its own with supportive care.
We refer to Wikipedia articles on medicine for help. The patient's 's listory of front seat unrestrained driver in motor vehicle collision and elevated pulmonary capillary wedge pressure and troponins suggest pulmonary contusion as the most likely diagnosis, Cardiac contusion or hemorrhage would have presented with additional symptoms such as ECG changes and ECG changes in different pat- tern, while Takotsubo cardionyopathy is not typi- cally assnciated with chest trauma or BCG chunges. Answer: C
(Ours, renakkrr)
the lung tissue.
the 1960s its
and airbags reduces
Gold (Oracle)
Pulmonary contusion pulmonary contusion, also known as lung contusion, is bruise of the lung, caused hy chest trauma. As a result damage to capillaries, blood and other fluids accumulate in the lung tissue. The excess fluid interferes with gas exchange, poten- tially leading to inadequate oxygen levels (hypoxia). Unlike pulmonary laceration, another type of lung in- jury. pulmonary contusion does not involve a cut or tear of the lung tissue. pulmonary contusion is usu- ally caused directly by blunt trauma but can also result from explosion injuries or shock wave associated with penetrating trauma. With the use of explosives during World Wars and II. pulmonary contusion re- sulting from blasts gained recognition. In the 1960s its occurrence in civilians began to receive wider recog- nition, which cases it usually caused hy traffic of seat belts and airbags reduces
We refer Wikipedia articles medicine for help. The patient's 's elevated troponins and pul- monury capillury wedge pressure, along with chest pain and bruising over the chest wall, suggests car- diac contusion, which can occur from blunt chest trauma like motor vehicle collision. Hemor- rhage would not cause elevated troponins and pul- monary capillary wedge pressure. Palmonary con- tusion would cause respiratory symptoms. Takot- subo cardionyopathy most likely diugnosis given the patient's 's history of trauma. Answer:
possibility. but not the
accidents. The the risk to vehicle occupants. Diagnosis is made by studying the cause of the injury, physical examination and chest radiography. Typical signs and symptoms include direct effects of the physical trauma, such as chest pain and coughing up bload, us well as signs that the body not receiving enough oxygen, such as eyanosis, The contusion frequently heals on its own with supportive care.
28

---

Table 18: SA Example of Failure Case (first category). We compare the generated output from the Flan-T5 XL model with KARD and reranker against the gold correct rationale from ChatGPT. The answer is in blue. We mark the hallucination in the generated rationale yellow and highlight the factually correct statements the generated rationale and its supporting fact in the passage as red.
Question: Could the Powepuff Girls make the background to the Azerbuijani flag? (A) yes (B)
Retrieved Passage
Outputs
KARD (Ours, renanerr
Flug Japen. . Two recently designed national flags re- semble the Japanese flag. In 1971. Bangladesh gained independence from Pakistan, and adopted a national flag that had green background, charged with an off centered red disc that contained golden map of Bangladesh. The current flag, adopted in 1972, Baedlee dropped the golden map and kept everything else. The Government of Bangladesh officially calls the red disc circle; the shed to create their country. The island nation of Palau uses a flag of similar design. but the color scheme is completely different. While Government of Pulau does not cite the Japanese flag as an influence on their national flug, Japan did administer Palau from 1914 until 1 1944, (..))
First. we need to know what background of the Azerbaijani flag is. The flag Azerbaijan has blue background with white text and blue ver- tical bands, Second, we need to know what the Powepulf Girls look like. As far as we know, they are fictional character created by the fictional net- work Cattoon Network. Bused on these two pieces of information, we say that the Powepulf Girls are not real country, and they do not have the ca- pabiliyy or knowledge to design flag background. The colors and design of the Azerhaijani flag unique and original, and there is no evidence that they have ever heen
color symbolizes the blood that was
as back for flag. Therefore. the answer the question is (B) no. the Powepull' Girls could not make the background to the Azerbaijani flag. Answer:
Gold (Oracle)
Flag of Azerbaijan The national flag of Azerhaijan consists of three horizontal stripes of equal width, from top bottom: light blue, red, and green. In the center are white crescent and eight-pointed star. The hasic description of the flag, along with the ratio, can be found in the Constitution Azerhaijan, which was adopted 12 November 1995:
1. Who the Powerpufl' Girls? They are three characters from un animated television series who have superpowers and fight crime. What Azerhuijani flag? The flag has three horizontal stripes: blue, red, and green In the center. there is white crescent and an eight-pointed star. Can the Powerpuff Girls make the background the Azerbuijani flag? The answer is yes. The back- ground consists three horizontal stripes, which the Powerpuff' Girls can easily create with their powers. They can make the stripes blue. red, and green in the correct order, It is not stated the question whether they can create the crescent and the center, but since question only asks about the background. we can assume that they do not need to create the symbols the center. There- fore, the answer (A) yes. Answer: A
29

---

Table 19: Strategy Example of Failure Case (xecond calegory). We compare the generated output from the Flan-TS XL model with KARD reranker against the gold rationale from ChatGPT. The answer in blue. We mark the hallucination in the generated rationalc as yellow and highlight the factually correct statements in the generated rationale and its supporting fact in the passage as red.
Question: Do human sacrums have more fused vertebrac than an Alaskan Malamute? (A) yes (B) no
Retrieved Passage
Outputs
KARD
Sacrum
The sacrum or plural: "sacra" or "sacrums") () in human anatomy is a large, triangular bone the base of the spine, that forms by the fusing of sacral vertehrae S1-S5 hetween 18 and years of age.
First, need to understand what the sacrum and how it is related fused vertebrae. The sacrum is triangular bone located the base of the spine in humans. Next, we need to understand how ver- tebrac are fused. In the human spine, there are six fused vertehrae (S1, S2, S4), while Alaskan Malamute, there only fused febrae (S3-54), Now, vertebrae in the human sacrum with those of the Alaskan Malamute. The human sucrum has fused vertebrae, while the Alaskan Malamute has five fused vertebrae. Therefore, the answer to the question (B) no, human
(Ours, renanker)
compare the fused
do have more fused vertebrae than an Alaskan Malamute. Answer:
Gold (Oracle)
Axial skeleton At birth the majority of humans have 33 separate vertebrae. However, during normal devel- opment several vertebrae fisse logether, leaving total of 24, or not there are 32 34 vertehrae stems from the fact that the two lowest vertebrae, the sacrum and the coe- eyx, single hones made up of several smaller bones which have fused together. This is how the vertebrae are counted: separate vertebrae and the sacrum, formed from fused vertebrae and the coceyx. formed from 3-5 fused vertebrae. If you count the coccyx and sacrum each one vertebra, then there are 26 verte- bree. If the fused vertehrae all counted separately, then the total number of vertebrae comes to between 32 34.
Firstly. what sacrum? A sacrum triangu- lar bone at the base of the spine
composed five fused vertebrae humans and many other mammals. Secondly, what is an Alaskan Mala- mute? It is breed of dog that is known for its strength and endurance. Now, the question is ask- ing whether the human sacrums have more fused vertebrae dog breeds tend to have mans. However, when comes to the specific the sacrum, the answer that humans have more fused vertebrae. As mentioned carlier, human sacrums consist of Alaskan Malamutes have four. Therefore, the cor- rect answer is (A) Yes, Answer: A
cases. The confusion about whether
Alaskan Malamute. In general, vertebrae than hu-
vertehrae, whereas
30


Reviews:
[{"review_id": "fYuOD1PZHs", "reviewer": "Reviewer_jTmB", "summary": "The paper deals with the challenge of utilizing small LMs in knowledge intensive tasks. As recent LLMs have shown promising capabilities in tasks that require reasoning, however, deployment of such model can remain limited due to cost or data limitations. Thus, the authors turn to face the challenge of reasoning distillation to other LMs, mainly smaller LMs that might be more feasible for deployment. However, small LLMs are limited by their inferior limited capacity of knowledge and understanding, therefore an external source of relevant context could be utilized for bridging this gap.\n\nThe authors present Knowledge augmented reasoning distillation (KARD) that fine-tunes smaller LMs to generate rationals, with the aid of a LLM with high reasoning capabilities, and to augment its knowledge with relevant contexts from an external knowledge base, to answer complex questions. KARD utilized baseline approaches to tackle knowledge intensive tasks, such as CoT prompting to the LLM and fine-tuning a neural document re-ranker to retrieve high quality contexts during inference when access to a large LLM is unavailable.\n\nThe authors test their system on two knowledge intensive benchmarks that require some level of reasoning using one of multiple sources and/or steps (MedQA-USMLE and StrategyQA). The evaluations show that using KARD with a 250M parameter model shows superior performance compared to fine tuned models, knowledge augmented and in few-shot settings. Furthermore, the authors analyze how model size, training set size and reranker setup affects the performance of KARD.\n\nThe main contributions of the paper are: (1) novel method that combines reasoning distillation with knowledge augmentation using neural rerankers. (2) analysis showing that small LM (250M) are not sufficient for knowledge intensive tasks in domain specific cases. (3) KARD's performance vs. various baselines and technique is superior.", "questions": "- The objective functions of the ranker and distillation (L_rerank and L_distill_kb) are independent. Why is that? have you evaluated combining both model updates?", "limitations": "The authors broadly addressed the major weaknesses and limitations of the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The challenge presented is of high importance for real-world applications and has high value. \n- Experimental setup, baselines chosen and evaluation results of KARD on MedQA-USMLE and StrategyQA are convincing and significant with clear accuracy improvement of smaller LMs (and also larger) compared to the different baselines. Proving the paper's main claim.\n- Improving the neural re-ranker using the LLM's rationals aids KARD to find relevant contexts during inference, and beyond naive approaches like BM25. \n- Well written with clear problem presentation, concept explanations, mathematical notations and definitions. It is easy to follow the concepts in the paper. ", "weaknesses": "- Limited evaluation on different datasets (only 2) and models (T5/Flan-T5). In addition, performance of KARD methods vs. reasoning distillation in the StrategyQA task is limited compared to MedQA. The authors address this in the limitations section.\n- A generic external knowledge base (wikipedia) that might not be suitable for medical-oriented tasks such as MedQA for example. We can further see evidence to this looking at the KARD (silver knowledge, oracle) evaluation where the gap between that model and with the best KARD is smaller vs. the large LLM (chatGPT). \n- Somewhat limited development (or naive approach) of the main contribution of the paper, that is the combination of neural ranker with reasoning distillation. To the authors credit they have addressed this in the paper. The authors address this in the limitations section.\n- Authors have addressed but did not evaluate a joint objective functions of neural ranker + distillation.\n- Despite authors addressing many of the paper's limitations, I find it significantly hinders the reliability of work done in the paper.\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in providing constructive feedback, and we address your concern and questions below. \n\n---\n\n> W1 & W6. Limited evaluation which significantly hinders the reliability of work\n\nWe perform additional experiments on another dataset OpenbookQA [1] and with another decoder-only language model (OPT [2]). As shown in the below table, our KARD outperforms other baselines in those experimental settings, showcasing its generalizability.\n\n||OpenbookQA | OpenbookQA | OpenbookQA | MedQA-USMLE |\n|:---|:---:|:---:|:---:|:---:|\n||T5-base| T5-large | T5-xl | OPT-350M |\n|Fine-tuning|54.0|62.0|74.6|26.47|\n|Knowledge-augmented FT|53.8|64.6|73.8|25.84|\n|Reasoning Distillation|58.2|65.8|76.2|28.67|\n|KARD (BM25)|55.4|65.4|75.6|31.26|\n|KARD (Reranker)|**59.2**|**66.2**|**78.6**|**34.25**|\n\n[1] Mihaylov et al., Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering, EMNLP 2018.\n\n[2] Zhang et al., OPT: Open Pre-trained Transformer Language Models, preprint 2022\n\n---\n\n> W2. The performance improvement is limited in the StrategyQA task compared to the MedQA\n\nWe believe the performance improvement in StrategyQA task is significant, considering that KARD enables small models to achieve performance comparable to models with rationales from ChatGPT, especially in Large and XL size models. \nThe performance gain may seem limited due to a smaller performance gap between the model without any knowledge augmentation and the Oracle model on StrategyQA compared to MedQA.\n\n---\n\n> W3. A generic external knowledge base (wikipedia) may not be suitable for the medical-oriented tasks. \n\nWe use Wikipedia as a KB in experiments due to its generalizability across diverse domains. Furthermore, since Wikipedia also contains in-depth information about the medical and general healthcare domain, using Wikipedia as a KB for medical-oriented tasks is suitable (reference: https://en.wikipedia.org/wiki/Health_information_on_Wikipedia).  Please refer to the example in Appendix Table 11, where the passage includes compound medical knowledge though it is retrieved from Wikipedia.\n\n\nMoreover, we experiment with PubMed as a knowledge base and show the results in the below table (initial experimental results already exist in Table 1 of Supplementary File). The results show that KARD with Pubmed outperforms standard reasoning distillation but underperforms KARD with Wikipedia due to the limited retrieval with BM25 on medical-domain passages.\n\nSpecifically, considering the huge gap between KARD with BM25 and KARD with silver knowledge performance, PubMed’s knowledge base contains highly informative passages. However, BM25 with a question as a query has a limited ability to retrieve them, which results in worse candidate documents being reranked than the candidate documents retrieved from Wikipedia.  This observation suggests a future research direction on the retrieval method tailored to the medical knowledge base.\n\n| Flan-T5 Base           | Wikipedia | PubMed |\n|---|:---:|:---:|\n| Reasoning Distillation |   31.03   |  31.03 |\n| KARD (BM25)            |   33.14   |  31.58 |\n| KARD (Reranker)        |   38.15   |  36.84 |\n| KARD (Silver knowledge, oracle)|   40.30   |  45.48 |\n\n---\n\n> W4. Somewhat limited development (or naive approach) of the main contribution of the paper.\n\nThe main focus and contribution of our paper is **our novel knowledge-augmented reasoning distillation and reranking methods** with their supporting theoretical and empirical results. \n\nIn particular, our main contribution is to **identify the issue arising in the reasoning distillation paradigm** due to the limited capacity of small language models to memorize knowledge. We propose an effective knowledge-augmented reasoning distillation to tackle the issue. Specifically, motivated by our theoretical analysis that knowledge augmentation with a retrieval reduces the amount of memorization to perform well on knowledge-intensive reasoning tasks, we retrieve a set of relevant documents that can guide the language model to correctly answer questions during the reasoning distillation. Lastly, we introduce a neural reranker to retrieve documents relevant to generating rationales that lead to answering correct answers.\n\n---\n\n> W5 & Q1. Reasons for independent training of small LMs and reranker, not joint training.\n\nWe would like to emphasize that we do not have to jointly train the reranker when distilling larger language models into smaller models. In particular, when distilling models with our method, we use the rationale that is generated from the large language model and the silver knowledge that is retrieved with the generated rationale (See Lines 187-190); therefore, the **reranking process is not involved during distillation**. On the other hand, our neural reranker is also trained separately by assigning high relevance scores to documents that are related to silver knowledge.\n\nMoreover, it is not straightforward to train the neural reranker along with the loss from distilling language models since reranking itself and choosing relevant documents are not differentiable operations. To enable it, we can alternatively utilize reinforcement learning algorithms [1], however, they are known to suffer from high variances. Also, if we consider retrieved documents as latent variables, we can approximate the marginal likelihood of training objectives with the retrieved top-k documents as done in the previous work [2]. However, since we need a large number of k different documents for better approximation of marginalization during training, it is computationally prohibitive to jointly train the retriever and the distilled language model. We leave addressing such technical challenges as future work.\n\n[1] Wang et al. R^3: Reinforced ranker-reader for open-domain question answering AAAI 2018.\n\n[2] Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020.\n\n"}, {"review_id": "DRvGcfGGCT", "reviewer": "Reviewer_RfCw", "summary": "In this paper, we propose the KARD model for small model Q&A through knowledge distillation + KB retrieval. The authors show experimentally that the model can outperform other models of 3B using only 250M parameters.", "questions": "1. What if KARD switches to use a medical KB for MedQA-USMLE?\n2. What is the time cost to train a reranker, as you need to compute the similarity between each pair of Wiki articles and training samples.", "limitations": "The authors have discussed potential technical limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. a model of LLM knowledge distillation + KB retrieval is proposed.\n2. KARD outperforms other fine-tuning models.\n3. a neural reranker based on similarity of rationales and passages is proposed.\n4. the paper is easy to read.", "weaknesses": "1. a problem of the paper is the experimental design. In Table 1, the authors do not compare other knowledge-augmented LMs. this leads to experimental comparisons that are inadequate and unfair.\n2. The finding of the paper-enhancing model effectiveness through KB retrieval-is not that surprising. This leads to a possible lack of innovation throughout the paper.\n3. for MedQA-USMLE, using wikipedia as KB may not be as effective as using specialized medical KB.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive comments. We faithfully addressed all your concerns and questions below:\n\n---\n\n> W1. Comparison against other knowledge-augmented LMs.\n\nThank you for your suggestion. Please note that our main argument is that **knowledge augmentation is important when conducting reasoning distillation** when finetuning small LMs, especially for knowledge-intensive reasoning tasks, to supplement the limited capacity of small LM for memorizing knowledge. Therefore, **a knowledge-augmented LM is not a direct competitor with our proposed method**. Rather, it is one of the possible base model that can benefit from reasoning distillation with our KARD. \n\nNevertheless, following your suggestion, we compared against Atlas [1], which is a state-of-the-art open-source knowledge-augmented LM, on the MedQA-USMLE dataset to measure the capability of knowledge-augmented LM on the tasks that require complex reasoning ability. In the experiment on the MedQA dataset, the Atlas-base model with 220M parameters shows accuracy of **31.03**, which is  **significantly worse** than our method **KARD** with accuracy of **38.15**. It implies that **Atlas has limited reasoning ability**, especially in the domain requiring expert knowledge, which emphasizes the importance of reasoning distillation. Motivated by our findings, incorporating reasoning distillation on knowledge-augmented LM can be a meaningful research direction for future work.\n\n[1] Izacard et al., Atlas: Few-shot Learning with Retrieval Augmented Language Models, 2022\n\n---\n\n> W2. The finding of the paper, enhancing model effectiveness through KB retrieval, is not that surprising. This leads to a possible lack of innovation throughout the paper.\n\nWe respectfully disagree with reviewer's point about the lack of innovation in our work because **this comment seems to be based on a critical misunderstanding of our work.** Our finding is **not limited** to enhancing model effectiveness through KB retrieval, and our method is not a straightforward combination of KB retrieval and reasoning distillation. Rather, our main contribution is in identifying the underrepresented issue that arises in the **reasoning distillation paradigm** due to the limited capacity of small language models to memorize knowledge, as clearly indicated in lines 42-43 of the main paper and identified by other reviewers (jTmB, 7NNq, AUtH).\n\nAlso, our method is motivated by our theoretical analysis, which shows that knowledge augmentation with retrieval reduces the amount of memorization to perform well on knowledge-intensive reasoning tasks, and we find that the knowledge augmentation during the reasoning distillation enhances the performance of small models in knowledge-intensive reasoning tasks, especially in the domains that require knowledge not included in general LMs, such as the medical domain. Furthermore, we also introduce a novel reranker to retrieve documents relevant to generating rationales that lead to answering correct answers.\n\n---\n\n> W3 & Q1. Using Wikipedia as KB may not be as effective as using specialized medical KB. What if KARD switches to use a medical KB for MedQA-USMLE?\n\nWe use Wikipedia as a KB in experiments due to its generalizability across diverse domains. Furthermore, since Wikipedia also contains in-depth information about the medical and general healthcare domain, using Wikipedia as a KB for medical-oriented tasks is not very limiting (reference: https://en.wikipedia.org/wiki/Health_information_on_Wikipedia).  Please refer to the example in Appendix Table 11, where the passage includes compound medical knowledge though it is retrieved from Wikipedia.\n\nMoreover, following your suggestion, we experiment with PubMed as a knowledge base and summarize the results in the below table (initial experimental results already exist in Table 1 of Supplementary File). The results show that KARD with PubMed outperforms standard reasoning distillation but underperforms KARD with Wikipedia due to the limited ability of BM25 to retrieve relevant documents from the medical domain corpus.\n\nSpecifically, considering the remarkable performance gain of  KARD with silver knowledge compared to BM25, the PubMed knowledge base is a valuable resource for knowledge augmentation. However, BM25 with a question as a query struggles to retrieve relevant documents from Pubmed's knowledge base. This observation shows a future research direction on the retrieval method tailored to the medical knowledge base.\n\n| Flan-T5 Base           | Wikipedia | PubMed |\n|------------------------|:---------:|:------:|\n| Reasoning Distillation |   31.03   |  31.03 |\n| KARD (BM25)            |   33.14   |  31.58 |\n| KARD (Reranker)        |   38.15   |  36.84 |\n| KARD (Silver knowledge) |   40.30   |  45.48 |\n\n---\n\n> Q2. What is the time cost to train a reranker for computing the similarity between each pair of Wiki articles and training samples?\n\nIn the training phase of the reranker, the initial step involves retrieving a set of candidate documents following the procedure outlined in lines 205-207 of the main paper, where the retriever takes about a second per a single training instance. Importantly, we can cache these candidate documents before the reranker training begins. As a result, there is no extra cost for computing similarity between all pairs of Wiki articles and training samples during reranker training.\n\nIn other words, we only need to load cached candidate documents for each instance in the minibatch. We summarize the wall-time for each operation in a single training iteration below to show that the time consumption for loading cached candidate documents is negligible compared to forward and backward computation for model training.\n\n|                    | data load | forward | backward |\n|--------------------|-----------|---------|----------|\n| Wall time (second) | 0.01      | 0.9     | 0.14     |\n| Ratio              | ~1%       | 85.7%   | 13.3%    |\n\n"}, {"review_id": "WRAAYcoJRX", "reviewer": "Reviewer_7NNq", "summary": "This paper proposes a retrieval-augmented knowledge distillation approach for QA tasks. This approach, KARD, extends reasoning distillation, which uses an LLM such as GPT-3.5 as a teacher model and distills a student model by learning from question and rationale pairs (generative loss). KARD has a retriever that obtains relevant documents based on the rationale. The retrieved documents are used for training a student model. In addition, a reranker is independently trained to select more relevant documents.\n\nThis approach is evaluated on two QA datasets: one from the biomedical domain, MedQA-USMLE, and StrategyQA, which covers more general and popular entities. EN Wikipedia is used as a knowledge base throughout experiments. The proposed approach, KARD (reasoning distillation+reranker), is compared with prompting approaches (e.g., few-shor, CoT) and other finetuning-based approaches (e.g., standard FT and knowledge-augmented FT). KARD outperforms all baselines by various margins depending on the datasets and model sizes. In addition, the authors provide analyses on several factors such as the number of rationales. \n", "questions": "- Related to the first point about weaknesses, I believe including an analysis of the datasets (for instance, entity types and the diversity of entities) could be helpful. This is because the two datasets seem sufficiently different, and these differences might explain the results. ", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work proposes KARD, which integrates retrieval and reranking modules into the distillation framework.  \n- The experimental results support the effectiveness of this approach particularly on the QA task from the medical domain. \n- Overall, this paper is well-written and easy to follow. The detailed analysis on model configurations is provided.\n", "weaknesses": "- Although this approach outperforms all the baselines, its gains become marginal with large model sizes, specifically in the cases of MedQA-USMLE and StrategyQA. On the other hand, KARD demonstrates a performance comparable to that of the oracle, ChatGPT, on StrategyQA, but lags significantly behind on MedQA-USMLE. The inconsistency in these results presents a challenge in interpretation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive and helpful comments. We initially address all your concerns and questions below:\n\n---\n\n> W1. Gains become marginal with large model sizes.\n\nAs stated in the main paper lines 108-110, memorization of training data is essential for achieving good performance in language tasks [1] and memorization capacity is proven to be proportional to the model size [2]. Thus, larger models with better memorization may less rely on the retrieved knowledge for answering questions, and thus gains from retrieval become small as the model size increases.  We empirically observe this pattern in our experiment and have clearly discussed it in lines 261-266.\n\nNevertheless, the largest XL model augmented with silver documents still outperforms the model without any knowledge augmentation in the MedQA-USMLE dataset. It implies that there is a potential of larger gains with better retrieval models, which opens up a promising direction for future work.\n\n[1] Brown et al., When is memorization of irrelevant training data necessary for high-accuracy learning?, STOC 2021.\n\n[2] Kim et al., Provable memorization capacity of transformers, ICLR 2023.\n\n---\n\n> W2. KARD lags significantly behind on MedQA-USMLE compared to ChatGPT but not much on StrategyQA. The inconsistency in these results presents a challenge in interpretation.\n\nWe think that such a discrepancy comes from different characteristics between MedQA and StrategyQA (See examples in Tables 11 and 12 in the Appendix). Specifically, the questions in MedQA are typically longer and contain more entities compared to the ones in StrategyQA, which requires a language model to leverage a significant amount of more knowledge to answer the questions in MedQA. Thus it is challenging for the smaller language model with the limited knowledge capacity to outperform larger language models like ChatGPT in MedQA, even when it is coupled with the knowledge retrieval. However, since the model augmented with silver knowledge achieves better performance than the one with knowledge retrieved by our reranker, we believe the gap between small and large language models can be further reduced with better retrievers.\n\n---\n\n> Q1. Regarding W2, including an analysis of the datasets could be helpful.\n\nThank you for your valuable suggestion. We include statistics of each dataset including the average number of entities and words in each question as follows:\n\n| |MedQA-USMLE | StrategyQA|\n|:---:|:---:|:---:|\n|Average number of entities | 35.72 | 1.19 |\n|Average number of words | 133.59 | 10.6 |\n\nAs we can observe, questions in MedQA-USMLE generally have longer questions having more entities than questions in StrategyQA.\n"}, {"review_id": "t69zc8RTN9", "reviewer": "Reviewer_AUtH", "summary": "The paper focuses on distilling the chain-of-thought reasoning capability from large LMs to small LMs in knowledge-intensive tasks. Since small LMs do not encode sufficient knowledge required for reasoning, the paper proposes to augment small models with a knowledge retriever that obtains relevant documents for a given task. Experiments show that the proposed method leads to more successful knowledge distillation, especially when the LM size is smaller.", "questions": "How do you obtain the rationales from the large LM? The chain-of-thought prompting asks the LM to generate the rationale and then the answer. Do you simply provide the answer to the large LM and ask it to rationalize the answer?", "limitations": "Limitations are well discussed in the paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe proposed idea is well-motivated and sound.\n2.\tSufficient experiments and detailed analysis are provided to demonstrate the effectiveness of the method.\n", "weaknesses": "One concern I have is the consequence of using multiple rationales to train the small LM, since this would misguide the model to learn that the answer prediction does not rely on the rationale. This may further lead to shortcut reasoning. I would suggest using simulation-based metrics to evaluate the faithfulness of the rationales to see if this is the case. Or you can randomly corrupt the generated rationales and see if the answer prediction is affected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive and helpful comments. We initially address all your concerns and questions below:\n\n---\n\n> W1. One concern: The consequence of using multiple rationales to train the small LM, since this would misguide the model to learn that answer prediction does not rely on the rationale. This may further lead to shortcut reasoning.\n\nThank you for your insightful comment. We have also noticed this problem to some extent, therefore, we first filter some erroneous rationales which lead to false prediction, following procedures described in lines 82-89 of the Appendix. In other words, if a small Flan T5 model makes a wrong prediction with the rationale generated by ChatGPT, such rationale is discarded. Furthermore, we have conducted analysis on the rationale diversity in Table 2 of the main paper. Corresponding results show that increasing the diversity of rationale (i.e., using multiple rationales) leads to better performance of small language models.\n\n---\n\n> Q1. How do you obtain the rationales from the large LM?\n\nWe use chain-of-thought prompting to generate rationales from the large LM. We explain the detailed procedure in lines 82-89 of the Appendix and provide the example prompt in Tables 4 and 5 of the Appendix.\n"}, {"review_id": "QkQfAn5Cky", "reviewer": "Reviewer_Z8sP", "summary": "The authors propose a form of knowledge distillation for a retriever-reader architecture. It uses rationales to guide the neural reranker to retrieve more relevant passages for reasoning, instead of passing the query to the retriever and retrieving the most similar passages. The paper includes an interesting set of ablations as well as some quantitative analysis and limitations. The generative process is as follows: at training time, a rationale is generated using a LLM. Successively, the rationale is passed to the retriever and it retrieves the top-k passages that are most similar to it Finally, a small LM is fine-tuned with the rationale, retrieved passages and question.\n\nAt inference time, the question is passed to the retriever, and top-k passages relevant to the question are retrieved (with BM25 plus a neural reranker which helps correct the initial ranking that is closer to the question than to the rationale). Then, a rationale is generated conditional con the reranked passages and input query.", "questions": "Please mention the encoder that is used to initialise the reranker in the main text, I have to tease it out of the appendix. \n\nWould it make sense to do a comparisons to augmentation with rationales plus filtering: Zelikman et al. (2022) STaR: Bootstrapping Reasoning With Reasoning or do we expect to always useful valid rationales?", "limitations": "The authors include a comprehensive potential societal impact section in the appendix.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality and significance: the authors' contribution is a nice application of both distillation and retrieval or knowledge augmentation for LMs.\nClarity: the paper is well written and it is clear and easy to read.", "weaknesses": "From the experiments section, it is not clear what type of retriever the baseline methods that include knowledge augmentation, is it a dense retriever or BM25? If it is the former, which specific encoder/decoder are used and in which task they are fine-tuned on? \nHere, I am assuming that the retriever is with BM25 in the baselines:\n\nSuch comparisons to more than one neural retrieval augmented language models are important to paint a full picture of the contribution. One possibility is to use NQ, TriviaQA, other Q&A evaluation datasets or the KILT datasets to understand whether both the retrieved passages and the rationale augmentation is useful for smaller LMs for simpler Q&A tasks besides the reasoning Q&A used in the paper. \n\nAlternatively, comparisons to other retrieval-augmented language models using the same benchmarks included in the paper can also help to quantitatively assess its performance grounding the claims that the cited paper mentions (i.e. that these models are not good for complex reasoning tasks). --> This experiment has been done during the rebuttal", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive and helpful comments. We initially address all your concerns and questions below:\n\n---\n\n> W1. It is not clear what type of retriever the baseline methods that include knowledge augmentation.\n\nThank you for pointing it out and we will include more details about the retriever in Section 5.1. As you mentioned, we use BM25 as the retriever for the baseline methods, as described in Section B of the Supplementary File.\n\n---\n\n> W2. Evaluation of proposed method on simple QA datasets (NQ, TriviaQA, KILT)\n\nThank you for your suggestion. However, we would like to emphasize that the primary target of our work is on reasoning distillation; therefore, we hope you understand that improving on the simple QA datasets, which may not require the complex reasoning ability of LMs, is neither our focus nor the scope of this work.\n\n---\n\n> W3. Comparisons to other retrieval-augmented language models using the same benchmarks included in the paper can also help to quantitatively assess its performance grounding the claims that the cited paper mentions (i.e., that these models are not good for complex reasoning tasks)\n\nThank you for your suggestion. \nWe respectfully emphasize that our main argument is that **knowledge augmentation is important when conducting reasoning distillation** for fine-tuning small LMs, especially for knowledge-intensive reasoning tasks, to supplement the limited capacity of small LM for memorizing knowledge. \nTherefore, a knowledge-augmented (retrieval-augmented) LM is not a direct competitor with our proposed method. \nRather, it is one of the possible backbone networks that we can use for reasoning distillation with our method KARD. \n\nNevertheless, following your suggestion, we compared against  Atlas [1], which is a state-of-the-art open-source knowledge-augmented LM, on the MedQA-USMLE dataset to measure the capability of knowledge-augmented LM on the tasks requiring complex reasoning ability. \n\nIn the experiment on the MedQA dataset, the Atlas-base model with 220M parameters shows accuracy of **31.03**, which is comparable to the fine-tuned Flan-T5 with 250M parameters but **significantly worse** than our method KARD with accuracy of **38.15**. This result implies that **Atlas has limited reasoning ability**, especially in the domain requiring expert knowledge, which emphasizes the importance of reasoning distillation. Motivated by our findings, incorporating reasoning distillation on knowledge-augmented LM can be a meaningful research direction for future work.\n\n[1] Izacard et al., Atlas: Few-shot Learning with Retrieval Augmented Language Models, 2022\n\n\n---\n\n> Q1. Please mention the encoder that is used to initialize the reranker in the main text.\n\nThank you for the suggestion. As described in lines 109-114 of the Supplementary File,  the encoders for MedQA and StrategyQA are initialized with BioLinkBERT-base and LinkBERT-base, respectively.  We will specify details about the reranker in Section 5.1 of the main paper.\n\n---\n\n> Q2. Would it make sense to do a comparison to augmentation with rationales + filtering as done in Zelikman et al. (2022)?\n\nThank you for the suggestion. We would like to emphasize that “filtering” is orthogonal to knowledge augmentation. We can apply any filtering method to our KARD. Furthermore, we have already done it in our experiments by removing wrong rationales generated from large language models following Ho et al. [2] (See lines 82-89 of Appendix).\n\n[1] Zelikman et al., Star: Bootstrapping reasoning with reasoning, NeurIPS 2022.\n\n[2] Ho et al., Large language models are reasoning teachers, ACL 2023."}]

Author rebuttal:
Dear Reviewers,

Thank you for your considerable efforts in reviewing our paper and providing insightful reviews to our work. We appreciate that reviewers find our proposed idea well-motivated and sound. 

We have responded to the individual comments from the reviewers below, and believe that we have successfully addressed all of them. To summarize our response,

- **[jTmB]** We include additional experimental results on one more dataset (OpenbookQA [1]) and a decoder-only pre-trained language model (OPT [2]).
- **[RfCw, Z8sP]** We discuss the relation of our work against a knowledge-augmented language model, including a comparison with Atlas [3].
- **[jTmB, RfCW]** We discuss the use of the medical knowledge base (PubMed corpus) as an alternative to Wikipedia for the MedQA-USMLE task.
- **[jTmB, 7NNq]** We include more discussions and analyses on the experimental results of Table 1.
- **[7NNq]** We conduct an analysis of the dataset statistics to compare the different features of both datasets used in experiments.
- **[RfCw]** We analyze and discuss the time cost to train a reranker.

We respectfully recommend the reviewer read our response and leave a comment if any concern or question still remains.

Sincerely,
Authors

[1] Mihaylov et al., Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering, EMNLP 2018.

[2] Zhang et al., OPT: Open Pre-trained Transformer Language Models, preprint 2022

[3] Izacard et al., Atlas: Few-shot Learning with Retrieval Augmented Language Models, 2022

Meta-review: Minimizing model size with knowledge augmentation is a key challenge in deployment of deep models.  Previous works [26, 15] have shown that reasoning distillation is less effective for knowledge-intensive reasoning tasks [13] where factual knowledge is important. This work proposes a method that fine-tunes small LMs to generate rationales with augmented knowledge retrieved from an external knowledge base. More specifically it combines reasoning distillation [13] with knowledge augmentation [14] using neural rerankers.

Experiment shows that small T5 and Flan-T5 models on the knowledge-intensive reasoning datasets (MedQA-USMLE, StrategyQA) outperforms a 3B (12 times larger) model.

Reviewers are happy with the motivation of the approach  (combining LLM distillation and retrieval), the clarity of the writing, and the significance of this work. They also feel that some limitations still remains
1. The KB is generic (wikipedia) not a more realistic domain specific KB.
2. The baselines do not include many ablation results making it hard to judge the exact contribution and novelty of specific techniques in this work.
Decision: Accept (poster)