Efficient Knowledge Injection in LLMs via Self-Distillation
Authors: Kalle Kujanpää
Pekka Marttinen
Harri Valpola
Alexander Ilin
Venue: Accepted by TMLR ()

Abstract: In many practical applications, large language models (LLMs) need to acquire new knowledge not present in their pre-training data. Efficiently leveraging this knowledge usually relies on supervised fine-tuning or retrieval-augmented generation (RAG). Although RAG has emerged as the industry standard for knowledge injection, fine-tuning has not yet achieved comparable success. This paper proposes utilizing prompt distillation, a self-distillation-based method previously explored primarily for style alignment and instruction tuning, to internalize new factual knowledge from free-form documents. Unlike prior methods, our approach requires neither larger teacher models nor structured knowledge formats. Across multiple LLM sizes and model families, we show that prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. We analyze the key factors contributing to prompt distillation's effectiveness and examine how it scales.

Published in Transactions on Machine Learning Research (08/2025)
Efficient Knowledge Injection in LLMs via Self-Distillation
Kalle Kujanpää
kalle. kujjanpaa@ualloo fi pekko, SaakaammattineeeBaaatoooo
Pekka Marttinen
Department of Computer Science, Aalto University Finmish Center for Artificial Intelligence (FCAT)
Harri Valpola
harri@systemmaa. com alezilin Ssystemmaisstmmmmmmmmm
Alexander Ilin
System 2 AI
Reviewed on OpenReview: https: // openreview. net/ net/forum? id= drYpdSnRJi
Abstract
In many practical applications, large language models (TLMs)) need to acquire new knowl- edge not present in their pre-training data. Efficiently leveraging this knowledge usually re- lies on supervised fine-tuning or retrieval-augmmented generation (RAG). Although RAG has emerged the industry standard for knowledge injection, fine-tuning has not. yet achieved comparable success. This paper proposes utilizing prompt distillation, self-distillation- based method previously explored primarily for style alignment and instruction tuning, to internalize new factual knowledge from free-form documents. Unlike prior methods, our ap- proach requires neither larger teacher models nor structured knowledge formats. Across mul- tiple LLM sizes and model families, we show that prompt distillation outperforms standard supervised fine-tuning and can even surpass RAG. We analyze the key factors contributing to prompt distillation's effectiveness and examine how it scales
92
90
RAG
PD RAG
88
PD (XL)
Correctniss (56)
86
Figure Average accuracy of responses across evaluation datasets. The proposed prompt distillation (PD) method performs competitively with RAG and surpasses supervised fine-tuning (SFT). Combining prompt distillation with RAG (PD+RAG) improves performance over standard RAG. Scaling up the fact coverage in PD training (PD XL) further enhances performance, outperforming all other methods.
Code available http::///ttttt.cccaaaaaeaaaes/es

---

Published in Transactions on Machine Learning Research (08/2025)
1 Introduction
In many practical scenarios, large language models must integrate new, domain-specific factual knowledge absent from their original pre-training corpus. Two main strategies address this challenge: fine-tuning and retrieval-angmented generation (RAG). Fine-tuning integrates knowledge into model parameters but is sensitive to prompt variations and prone to overfitting However, it helps reduce prompt length. In contrast, RAG excels at providing access to the most up-to-date information and is highly effective in responding to detailed queries, but requires an external knowledge base and can lead to very long prompts (Schick et al., 2024
Lewis al.) 2020).
RAG has bocome the industry standard for knowledge injection, demonstrating robust performance in various contexts CCenn et al.
2024). However, knowledge injection through supervised fine-tuning (SFT) has not yet
achieved comparable success, as replicating RAG's performance has proven challenging (Ovadia et al. 2023
Mecklenburg et al. 2024 Recent approaches for injecting knowledge via SFT typically involve superviss vised
learning using question-anssen answers generated by prompting expert models like GPT-4 (Achiam et al. 2023
or Claude (Anthropic 2024) using the new factual content. Since fine-tuning trains the student model
to replicate expert answers exactly with answer tokens as targets, data quality is crucial. Consequently, using the best available model for data generation is common practice. Given that modern LLMs already encode vast amounts of knowledge, fine-tuning should, in principle, be capable of efficiently integrating new information, but current methods fall short. This paper seeks to close the performance gap between RAG and fine-tuning in knowledge injection tasks.
Classical fine-tuning has several challenges. First, using a different model for answer generation may intro- duce a mismatch between the expert and student answering styles (Gudibande et al.) 2023). As a result, training may focus more on mimicking the answering style rather than learning the new factual information. Additionally, an intelligent expert may generate complex questions requiring high intelligence to answer, which the student may not possess. This issue was discussed, for example, by Mirzadeh et al. (2020) and Mitra et al. (2023). who carefully adjusted the prompts to allow a smaller model to learn more efficiently. Another concern arises from using tokens as targets since the same question can have multiple valid answers. Training a model to replicate the expert's answers verbatim risks overfitting, and the student model may not gencralize well to answer new questions. This issue could be potentially mitigated by sampling multiple responses from the expert model. However, this makes the training data generation process much more expensive, particularly when using large expert model or human-generated responses.
We explore prompl distillation, self-distillation approach, to mitigate these limitations. Prompt distillation involves generating synthetic inestion-answer pairs about the new factual content using the LLM itself. It leverages the concept of self-distillation, where the student model learns from distributions over answer tokens produced by the teacher model, which additionally receives the new knowledge in its prompt (Fig. D By using the same model as both teacher and student (with a LoRA adapter (Hu et al. (021 )) we eliminate style and capacity mismatches. Consequently, training focuses specifically on learning factual knowledge, avoiding issues like verbatim memorization or unintended imitation of an expert model's answering style.
No supervised annotations or structured formats are needed.
Prompt distillation has previously been explored primarily for stylistic alignment and instruction tuning. For example, Askell et al. (2021) used it for distilling alignment, [nnell et al. (2022) for detailed task instructions, and Choi et al. (2022) for learning to act according to a persona. QQ et al. (2024) used a related approach for knowledge editing, relying on larger expert models and structured triplcts. In contrast, our method
applies self-distillation directly to unstructured, free-form documents without supervision, enabling efficient internalization of complex factual knowledge while remaining independent of larger models.
We conduct extensive evaluations with the Llama-3 (Dubcy et al. 2024 and Qwen2.5 (Yang et al. 2024 model families on custom datasets derived from Squadshifts Miller et al. 2020) and the multi-hop HotpolQA benchmark (Yang et al., 2018). Our findings show that prompt distillation significantly surpasses standard supervised fine-tuning for knowledge injection and reasoning. On Squadshifts, its closod-book performance can match that of open-book RAG. In the more complex HotpotQA, RAG is a strong baseline. However, PD improves closed-book performance substantially and, more importantly, enhances RAG when used together.
2

---

Published in Transactions on Machine Learning Research (08/2025)
Q: How many gold medals did A: France won 16 gold medala in the last Olympics, which were held in Paris in 2024.
Student
LLM
TT (a:14: aI..i-1)
adupuse
[privileged info] Paris 2024 Olympic Medal Table.
KL
Q: How many gold medals did France win the last 0lympica? A: Prance won 16 gold medals in the last Olympics, which were held in Paris in 2024.
Teacher
LLM
To (as lc, q, a...i-))
Figure 2: An overview of the prompt distillation approach. The privileged information e in the teacher's prompt is distilled into the student's weights.
We summarize our main results from Squadshifts and compare prompt distillation to supervised fine-tuning and RAG in Figure D
This work highlights prompl distillation as a powerful yet under-explored method for injecting knowledge into LLMs without structured supervision or external retrieval. Our main contributions are: (1) introduc- ing prompt distillation to internalize knowledge from free-form documents, (2) validating its effectiveness across multiple LLM families and sizes, and (3) analyzing key factors, including distillation and data gener- ation temperature, the use of larger models as experts (for data generation) and teachers (providing target distributions), training data scale, and LoRA adapter size.
2 Prompt Distillation
Prompt distillation closely resembles imitation learning, where an expert model with additional knowledge e generates demonstrations for the student to learn from. knowledge injection, these demonstrations can be question-answar pairs (q, a), both of which are multi-token sequences. Classical imitation learning (such SFT for LLMs) restricts the student to learning from the expert's actions (Lokens a;) without accessing the expert's underlying policy. In contrast, prompt distillation uses open-source teacher LLM to expose the complete policy aa(a:cc, q. a). By processing the entire sequence containing q, and in a single pass with causal masking, we obtain the teacher's logits at every token position, giving us the complete policy. This full policy from the teacher contains significantly richer information than isolated tokens alone. The goal prompt distillation is thus to train a student model to approximate the teacher's full policy, but crucially, without receiving contect e in its prompt. Specifically, the student's policy should satisfy:
To. (a:lg, a1. no(aile, 01.5-) = DKL(mooas q, 11. (a, q. a1..-1)) = 0
2.1 Data Generation
The first step of prompt distillation is the generation of expert demonstrations, which set of question- answer pairs (q, a) about the new knowledge e. We first generate questions q using an LLM with a high temperature (7 > 1) to ensure that the questions varied and that as few possible are duplicated. Next, generate answers using LLM with high temperature (T 1) to encourage diversity. While this temperature is not high enough to produce frequent nonsense, it can occasionally introduce noisy or corrupted passages. This mild corruption does not harm training because the generated answers serve as inputs (not direct targets) for the student and teacher models (see Fig, 2 Crucially, the teacher's logits remain well-defined at each step, allowing the student to learn how to correct sub-optimal tokens if the teacher is capable of that. By distilling the teacher's policy ne (aylc, q, a1.;-1), the student learns stay on the right. track, which is the underlying idea of the DAGGER algorithm (Ross et al. 2011
3

---

Published in Transactions on Machine Learning Research (08/2025)
Since want to copy the teacher's policy, need to sample questions q and answers where has significant impact on the answer-generation policy To(a; c, q, a1..i-).. Naturally, this impact is greatest when the questions and answers are related to C. However, having some questions and answers only distantly related to c does not hurt; it merely reduces efficiency. In our experiments, we generate expert demonstrations in two steps: first, we generate the questions by either the model being fine-tuned or a specialized question- gencrating model (Nayak et al., 2024), and then we generate the answers either using the model itself or a larger model, to simulate the distilaation of more powerful expert.
2.2 Distillation
For the teacher model, we formulate a prompt consisting of the knowledge c, a question q, and an answer a. The teacher model computes logits 2i,v for every position i and token v in the vocabulary V in the answer a. The student model only receives the question q and answer as input and computes the output logits = We perform knowledge transfer at high temperature, following the classical knowledge distillation approach (iintnn et al. 2015). That is, the logits are converted into probabilities using the same temperature T for both the teacher's and student's distributions:
exp(21e/ To q, a1..)-)) Ene (xp(2i..//T)
exp(=f,u/T)
Fe (vlq, a1.i-1) En exp(<
The loss is the average KL divergence between the two distributions for all answer positions = 1..... NA:
NA C
NA
Tolule, q, a1.. log To (u q a1...-)) To (ulq. a
Σ (i) = NA NA
(1)
VEV
where NA is the number of answer tokens. The expected KT, divergence between the teacher and student is equal to the mutual information between e and aj, where the expectation taken over the distribution of c:
Eg. Ee[DKL (To(as) c. q. a1. || (as q, a1..)))) = I(c; a, (1..)))))
Mutual information measures the reduction in uncertainty of one random variable when the other is known. By aligning the student's distribution (a q, a1...-1) with the teacher's, we minimize the expected infor- mation gain at inference time that would come from seeing c, ensuring the student internalizes the extra information from the context during training. A graphical illustration of the distillation is shown in Fig. D
We distill the knowledge at high temperature T > 1, making the student focus more less probable tokens. That is, the student learns to actively avoid the answers that the teacher is avoiding, To achieve the same effect using the cross-entropy loss with one-hot targets, one would need an enormous mumber of samples to realize that some answers must have a low probability. We briefly explored incorporating mid-layer activations in the loss but found that using the output logits alone was simpler and efficient enough.
2.3 Comparison to Fine- Tuning with Hard Targets
Cross-entropy loss with hard (one-hot) labels forces student to assign all probability mass to a single token, potentially disrupting its existing distribution, even when it already aligns well with the teacher's knowledge. As result, the student needs many training samples to average out these disturbances and approximate the desired distribution. Furthermore, these disturbances not only require the student to process many training samples to approximate the desired distribution but also increase calastrophic forgetting, as the forced alignment with one-hot labels can override previously learned knowledge (Kirkpatrick et al. 2017).
KL divergence loss with soft targets lets the student retain its distribution where the teacher adds no new information, leading to more efficient learning. In contrast, cross-entropy loss with hard labels treats the answer tokens generated by the expert as ground-truth targets. This requires sampling the answers at a low temperature to produce factually correct outputs. However, this can result in less diverse answers and a higher degree of overfitting to these specific targets during fine-tuning. Consequently, the finetunnd student model may struggle to answer questions formulated differently.
4

---

Published in Transactions on Machine Learning Research (08/2025)
2.4 Student Model
In experiments, the student model is constructed to be the same as the teacher model, additionally equipped with LoRA adapter (Hu et al. 2021
8' = - A0,
where A is LoRA initialized to AO 0. Thus, the two models are identical at the beginning of training. During training, the adapter's weights 2O change as the new knowledge e gets distilled into the student's weights 0' The teacher remains unchanged. Our implementation uses the same network for both the student and the teacher, loggling the LoRA adapter to switch between roles. Thus, having two roles during training does not lead to increased memory consumption compared to SFT with a LoRA adapter. To save computing resources at the cost of storage, the teacher logits can be stored during answer generation to climinate all additional compute during fine-tuning compared to SFT with hard targets. Alternatively, if multi-epoch training is performed, the logits can be computed and saved during the first epoch for subsequent epochs.
2.5 Regularization
Generally, we are not interested in only injecting knowledge into model. We want the model to remain usable for other tasks and prevent overfitting. This can be easily achieved in prompt distillation: assume a dataset of instruction-response pairs, and in particular, an instruction and a response r. Assuming that the instruction-responee pair is unrelated to all the injected knowledge C, we want the student's response to remain unchanged during the fine-tuning. In practice, since domain-speeific data usually differs from the general instruction luning set, we treal these sets as disjoint; any rare overlap is negligible and can safely be ignored. We add the following KL divergence between the student and teacher to the loss function:
1
NR DKL-Ieg (i)
NR NR
1
To (1. 11..-1)
Creg
Σ
To(uli, log Te (1/ i, T1.3-1)
(2)
NR
where NR is the number of tokens in the response sequence. Minimizing this KL divergence forces the student's output distribution to remain close to its initial distribution for generic instructions, thus preventing catastrophic forgetting while allowing the model internalize the new knowledge.
3 Related Work
Prompt Engincering and In- In-Context Learning enable users to guide model behavior through instruc-
tions and examples (Brown el al. 2020 Sanh et al., 2021 Lewis et al. 2020 Liu et al. 2023). Retrieval- augmented generation (RAG) extends this by integrating externally retrieved knowledge into prompts (Lewis) let al.) 2020 Izacard & Grave 2020). However, longer, complex prompts make it harder for LLMs process and integrate relevant information into their reasoning effectively. Prompt-based knowledge is transient, as it disa disappears once the prompt changes. Despite increasing context lengths, LLMs remain limited in managing extensive information within a single prompt (Liu et al. 2024b).
Unsupervised Fine-Tuning Tuning (UFT) of LLMs has been used for knowledge injection with mixed results. Training T.LMs on next-token prediction tasks on new documents can lead to performance gains as large language models learn memorize their training data (Gururangan et al. 2020 Carlini et al. 2021 2022). However, UFT has often underperformed in knowledge injection compared to RAG, particularly in Lasks involving new factual information. Ovadia et al. (2023) highlight LLMs' challenges in learning new facts through UFT, while Nayak et al. (204)) show it can sometimes reverse prior instruction tuning gains.
Supervised Fine-Tuning for knowledge injection proceeds by training pre-trained LLMs on tasks derived from new documents, either generated by standard LLMs (Mecklenburg et al. 2024 or specialized models Nayak et al.] 2024). RAFT (Zhang et al.] (2024) integrates RAG with SFT to filter irrelevant information and prevent performance degradation due to distractors (Shi et al. 2023a Mallen et al.) (022).. Liu et al. (0244a propose two-stage approach consisting of continual pre-training and SFT. Gupta al. (0244)
5

---

Published in Transactions on Machine Learning Research (08/2025)
show that fine-tuning and RAG can be combined to maximize domain-specific performance. Unlike RAG, fine-tuning can enable cumulative knowledge integration without increasing prompt. length, allowing models to improve on complex tasks (Snell et al., 2022
Choi et al. 2022: Ye et al. 2023 Liu et al. 2024a
Knowledge Distillation is a technique where a smaller model learns from a larger model, first introduced
by Hiinton et al. (2015). Tt has been widely applied in NLP (see, e.g. [Sanh et al. 2019 In the context of LLMs, distillation has been used to transfer the abilities of larger, proprietary LLMs smaller models
otherwise improve the performance of smaller models with the larger teacher model (Peng et al.) 2023
Taori et al.
2023 Chiang et al., 2023 Xu et al. 2023a b Mukherjee et al.) 2023 Recent advances refine
teacher prompts to clicit more detailed responses, including reasoning explanations (Mukherjee et al. 2023
Mitra et al.
2023)) luned the teacher's prompt to better align with the student's 's learning needs. Research by Wang et al. 2020 and Mukherjee & Awadallah (220)) suggests that utilizing richer signals, such as logits,
intermediate representations, and attention states, can enhance the distillation process.
Self-Distillation allows model to learn from its own outputs, performing sell-improvement. The teacher can access privileged information, such cleaner inputs, while the student model learns from noisier incomplete data. This technique has been particularly effective in semi-supervised learning (see, e.g., Tar- vainen & Valpola, 2017 and He et al. (2019)). Self-distillation is particularly relevant to our work, as it eliminates the need for a more advanced teacher model, simplifying the distillation while eliminating the initial mismatch between the teacher and the student.
Context Distillation has been explored in prior work, including Askell et al. (2021) ; which distilled human- aligned innvessaaionnl examples into model weights to improve answering style rather than factual knowledge, and [nnel ct al. 2022), which demonstrated distillation of detailed task instruction and reasoning steps. Choi et al. (2022) distilled brief instructions and persona-defining prompts and trained separate model to generate distillation data, whereas we show that standard LLM prompting alone suffices for learning, achieving competitive performance with RAG. Qi et al. (2024) applied context distillation for knowledge editing, representing the knowledge to be edited as structured triplets and leveraging large expert models. In contrast, we focus on a different problem, knowledge injection from ructuued documents, and employ self- distillation, making our approach independent of larger models while often achieving superior performance.
4 Experiments
We evaluate prompt distillation for knowledge injection using& a dataset derived from Squadshifts (Miller et al.) 2020). The original dataset was designed for an open- book setting, where the model retrieves answers from a provided document. The dataset consists of a pas- sage of text, question, and spans of text within the passage that contain the answer. We fine-tune mod- els for a closed-book setting, where answers must be generated without the document in the prompt. Since many original questions lack sufficient context to be un- Reddit
Table 1: Summary of the dataset derived from Squadshifts
Dataset
Documents
Number of Tokens
Amazon
207
363,031
New Wiki
203
252,035
NYT
188
338,918
209
368,214
derstood without the document, we reformulate them while prescrving the original answers. We use Llama-3-70B-Imm Instrructt to generate reformulated questions, which constitute our test set for evaluating fine-tuned models.
The test set includes 1,000 questions from each Squadshifts variant: Wikipedia, New York Times articles, Reddit posts, and Amazon product reviews. The mumber of passages used corresponds to the documents for the first 1,000 questions, ranging from 188 (NYT) to 209 (Reddit) (see Table []. We perform experiments on the four individual subsets separately. To ensure a valid evaluation, test questions must probe knowledge not alrcady known to the basc model. To test this, we evaluate the performances of the base models on All prompts used for question generation, answer
the test questions (see the base model results in Table 2.
generation, and grading are presented in Appendix O
6

---

Published in Transactions on Machine Learning Research (08/2025)
4.1 Implementation of Prompt Distillation
We evaluate prompt distillation on three instruct-tuned models from different families and sizes: Llama-3-81 Qwen2.5-14B, and Qwen2.5-3B. We generate 30 training questions for each test question using the evaluated model, sampling at a high temperature (1.5) to ensure diversity. Training questions are generated solely from the source documents without access to test questions. We do not explicitly prevent training questions from resembling test questions, but any similarity is incidental, as test questions are reformulations (produced by different model) of the originals, which the training question generator never accessed. For each evaluated 1.5.
model, we generate answers to the training questions using the model itself with temperature
We use the training questions to fine-tune each model using the prompt distillation approach. The student model uses a LoRA adapter, with rank 1024 for the 3B and 8B models and 512 for the 14B model, applied to all layers. We train all models using AdamW with a learning rate of 10-5, linear LR warmup, and a batch size of 4 per GPU. We fine-tune the 8B model on one AMD MI250X GPU for 24 hours (* 10 epochs). The 3B model is trained on one GPU and the 14B model on 8 GPUs for five epochs. In initial experiments, we exclude regularization due to its added computational cost. At test time, present each test question individually to the fine-tuned model, sampling an answer with a temperature of 0.25. For the complete set of hyperparameters for prompt distillation, please see Table S in Appendix D
4.2 Answer Grading
We use LLM-as-a-judge grading (Zheng et al. 2023 relying on 1aama---8B-Instruue with Chain-of- Thought-like prompt. The grading prompt follows two steps: first, the model generates a justification for the grade; then, it assigns the grade itself. Grades are binary: true or false. This two-step prompt significantly improves grading accuracy, as encourages the model to reason before deciding, similar to a thoougtf process. Without explicit reasoning, the model struggles to produce reliable judgments. Manual inspection of test subset estimated that approximately 2 % of grades were clearly incorrect (sce Appendix Dp. To further validate grading accuracy, we re-evaluated selected experiments using Qwen2.5-32B- Instruct (see Appendix M. The choice of grader had minimal impact, except in the case of non-fine-tuned instruct models without RAG. The conclusions remain unchanged. Alongside LLM-as-a-judge, we used sub- string match grading, following Mallen et al. (2022). Here, an answer is marked as correct if any of the gold answers appear as a substring in the generated response after simple normalization. The detailed results and a discussion of this metric are presented in Appendix(N] This alternative metric confirmed the same relative performance trends observed with our primary LLM-based evaluation.
4.3 Baselines
Supervised Fine-Tuning (SFT). We use the same 30 training questions as in prompt distillation. How- ever, for SFT, the answers are re-sampled at lower temperature (0.25) to reduce variance in responses. We fine-tune the models using LoRA adapter and optimize the standard cross-entropy token loss. All other training parameters, including duration, infrastructure, and learning rate, remain consistent with prompt distillation. We fine-tune models using both their own generated answers and those from larger models, simulating API-based distillation without access to logits.
RAG. We test the performance of the instruct models with RAG. We employ two different retrieval tech- niques: BM25 and embelding-based. For BM25, we create a document database by tokenizing all context paragraphs from the evaluation dataset with the Llama-3 and Qwen tokenizers. We then tokenize the ques- tions using the same tokenizer and use the Okapi BM25 ranking function (Robertson et al., 1995) from the rank-bm25 Python library to perform retrieval. In the embedding based approach, we embed all the documents and questions with OpenAl's Embedding-AAP API, using the Real--embedding-----mlll model. We use cosine similarity to retrieve the most relevant documents. For RAG, we retrieve the & = 7 most relevant documents per query and append them to the prompt. The value of & maximized within Llama-3-8B- Instruct's context limits. In our preliminary tests, we found that the performance increases with a larger number of documents in the prompt. We further tested RAG in combination with fine-tuned models.
7

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: The average answer correctness (%) on the question answering task the closed-bood (upper part)
and RAG (lower part) scenarios. The uncertainty is standard errois of the mean.
Base Model: Llama-3-8B- -B--urtuuct
Method
Amazon
New
Wiki
NYT
Reddit
Prompt Distillation
86.1
+0.2
94.4
+0.3
93.6
=0.6
79.5
+1.4
Supervised Fine- TaninT
75.9
+1.8
89.5
+0.2
87.5
=0.2
69.8
#0.4
Unsupervised Fine-Tuning TuninT
39.5
+1.6
63.1
+0066
52.6 =1.4
30.9
+2.7
Llama-3- B-S-unstrutt
22.1
61.2
38.2
20.8
Prompt Distillation + RAG
88.5
+0.3
96.7
#0.2
96.9
=0.2
77.8
#1.
SFT w/ Distractors + RAG
87.0 #0.7
95.1 #0.4
94.2 =0.1
82.4 +0.8
Llama-3 Maama-3-BB-ae 8B-Instruct + RAG
86.3
95.6
96.3
78.6
Base Model: 1ree2.55-4BBInstreee
Method
Amazon
New
Wiki
NYT
Reddit
Prompt Distillation
85.5 10.6
94.9 #0.3
92.6
+0.8
77.6 =0.4
Supervised Fine-Tumine
78.2 #1.7
90.2 #0.1
87.1
#0.6
71.5 #0.8
Owen2.5-14B-Insiee
16.5
61.9
31.2
16.7
Prompt Distillation + RAG
88.4 +0.4
97.3
+0.1
96.3
+0.4
83.6 +1.3
SFT + RAG
87.9 +0.3
96.6 +0.6
95.7 #0.6
83.1 #0.3
Qwen2.5-11B-Instruud. RAG
87.1
97.1
94.8
81.3
Base Model: Qwenn.553B-IIntrruc
Method
Amazon
New Wiki
NYT
Reddit
Prompt Distillation
76.4 +1.5
90.0
+1.3
84.6
+03
65.6
+0.6
Supervised Fine-Tuning
69.1 +0.7
84.3
#1.
76.6
=0.8
59.8 +0.5
Qwen2.5 5-3.2..-----m- 3-3B-Instruct
12.6
46.7
18.6
12.1
Prompt Distillation + RAG
86.5 +0066
94.8
+-0.6
95.0
- 0.3
75.6
+1.0
SFT + RAG
81.7 +0.8
92.2
+0.9
89.6 =1.0
68.7 +0.9
Qwen2.5-3B-Instree + RAG
85.4
94.5
91.9
75.3
Supervised Fine-Tuning with Distractors. We implement an SFT method inspired by RAFT (Zhang let al. 2024). During fine-tuning, two random distractor documents from the same domain are appended to the context alongside the golden document. The order of the documents is randomized to prevent the model from identifying the correct document based solely position. To improve robustness, the correct snippet is omitted with a 40% probability. This ensures the model learns to answer questions even when the retrieved context lacks the correct document, aligning with the optimal setting in Zhang et al. 2024 (see their Fig. 5). This approach combines elements of both open-book and closed-book fine-tuning. Training data generated using a Chain-of-Thoughts prompt., as emphasized the original work.
This setting mirrors the base model pre-training, aiming to predict the next token. The paragraphs sampled from the documents in the Squadshifts datasets by chunking (splitting into smaller snippets) while keeping overlap, ensuring context from one snippet carries into the next. Three epochs provided the best performance after testing different training durations and learning rates.
Unsupervised Fine-Tuning.
All fine-tuned models use the same training questions, LoRA adapter, and training durations to match the PD sctup, except unsupervised fine-tuning, where longer training causes overfitting and no training questions are utilized. Retrieval methods (Okapi BM25, OpenAI Embeddings) and retrieved snippet count remained constant across methods. Experiments used three seeds, except for the base I.I.M, which not re-trained.
8

---

Published in Transactions on Machine Learning Research (08/2025)
Table 3: The average answer correctness (%) prompt distillation (PD) and SFT with different-sized Llama-3 (8B & 70B) and Qwen2.5 (3B, 14B & 72B) models generating the answer tokens (expert) and target distributions (teacher). The top row for each subtable is the standard PD configuration. The uncertainty is two standard errors of the mean.
Base Model: Llama-3-8B-Instrucc
Method
Expert
Teacher
Amazon
New
Wiki
NYT
Reddit
PD
8B
8B
86.1
+0.2
94.4
+0.3
93.6
+0.6
79.5 11.4
PD
70B
8B
84.8 =0.4
94.3
#0.2
93.2
#0.2
78.4
=0.2
PD
70B
70B
84.0 +0.5
93.0
+0.5
92.3
#0.1
82.0
=033
PD
72B (Qwen2.5)
8B
84.8 #1.5
93.2
#0.3
92.3
#0.5
78.2
=1.2
SFT
8B
75.9 +1.8
89.5
+0.2
87.5
+0.2
69.8
0.4
SFT
70B
75.6 =1.0
89.2
+0.9
85.6
11.
70.9
=03
SFT
72B (Qwen2.5)
74.2 =0.5
88.7 +12
84.0
#0.6
60.5 =2.3
Base Model:
@wen2.5-14BB Instruct
Method
Expert
Teacher
Amazon
New Wiki
NYT
Reddit
PD
14B
14B
85.5
#0.6
94.9 10.3
92.6
10.8
77.6
=0.4
PD
72B
14B
85.1 20.5
94.3
10.4
92.4
10.5
76.6
10.9
PD
72B
72B
81.7 #0.6
93.6
#0.6
90.8
#0.6
69.9
#0.2
SFT
14B
78.2
+1.7
90.2
+0.1
87.1
+006
71.5
+0.8
SFT
72B
76.8 +0.9
90.0 #0.5
86.3
110
68.7
10.5
Base Model: Qwen2.5-3B-Instrue
Method
Expert
Teacher
Amazon
New
Wiki
NYT
Reddit
PD
3B
3B
76.4
+1.5
90.0
+1.3
84.6
10.3
65.6
10.6
PD
14B
3B
78.4
10.6
90.8
+0.2
87.3
+0.6
69.5 +0.6
SFT
3B
69.1
=0.7
84.3
#1.4
76.6
10.8
59.8
+0.5
SFT
14B
74.3
+1.0
87.6
+1.2
85.3
+1.7
67.7 #0.6
4.4 Results
Results without Larger Models. In Table 2 we evaluate the performance of Prompt Distillation (PD) using three different base models, Mhama-3-8B-Intree and compare PD to the baselines under two settings: plain question-answering scenario a closed-book setting (upper part of each table) and question-answering enhanced by RAG (lower part). For Llama-3, PD is the strongest method among the four non-RAG approaches, significantly outperforming supervised fine-tuning and achieving rAa performance. In contrast, unsupervised fine-tuning performs poorly. With RAG, PD achieves the highest overall performance. Notably, despite no explicit fine-tuning for RAG. PD effectively leverages retrieved context without disregarding it.
1enm2..-3B-Instruee and Qwen2.5-144-Innste
SFT with distractors achieves the best RAG performance on the Reddit dataset. Retrieving and attending the correct document on Reddit proved particularly difficult, as many posts follow similar patterns, often socking help or advice, making easy to confuse relevant and irrelevant passages when answering questions. Adding distractor documents during supervised fine-tuning enhances robustness misleading retrieved context but reduces closed-book performance without RAG. This is orthogonal direction to PD and could potentially be combined for an even higher potential performance gain in the RAG setting.
With the Qwen2.5-fann -ammily models, we compare PD to SFT, the strongest baseline with Llama-3-8B-Inse Instrrct... We also compare PD to the instruct model, both without R.AG and with OpenAl Embeddings based RAG, as outperformed BM25 with token-basod emboddings. For Qwen2.5-144 1-BBB, PD outperforms SFT with and without RAG and improves on the instruct model when combined with RAG. Across all datasets, the performance gap between PD and instruct RAG is smaller than between SFT and PD. The same conclusions

---

Published in Transactions on Machine Learning Research (08/2025)
Table 4: The average answer correctness (%) of prompt distillation and supervised fine-tuning with different temperatures, when the training data has been generated by Llama-3-8B Instruct. The top row is the standard prompt distillation configuration. The uncertainty is two standard errors of the mean.
Method
Amazon
New
Wiki
NYT
Reddit
PD
2
1.5
86.1
+0.2
94.4
+0.3
93.6
+0.6
79.5
+1.4
PD
1
1.5
83.8
#0.9
93.3
+0.5
91.8
#0.2
78.2
#0.9
PD
2
0.25
83.6
+1.1
93.9
+0.1
92.7
+0.5
79.0
#1.
SFT
N/A
0.25
75.9
=1.8
89.5
+0.2
87.5
#0.2
69.8
=0.4
SFT
N/A
1.5
75.0
#0.2
88.8
+0.8
85.1 #1.1
64.1 #1.9
hold for the 3B model, where PD remains the strongest method in both settings, consistently surpassing the non-fine-tuned RAG baseline.
Almost all differences between PD and baseline methods exceed the two standard error confidence intervals. Instruct are provided in Appen- dices P and Q Appendix E presents complete RAG results, comparing BM25 and OpenAT's AmP.ddinggAPI. We also include Oracle retrieval results to quantify the performance loss from imperfect retrieval.
Qualitative examples of PD with
and Qwen2.5-145 Ins-ruuutt
Distillation from Larger Models. Traditional distillation is commonly used to transfer knowledge and reasoning abilities from a larger model to smaller one. In contrast, PD employs self- distilltiinnnnn where the student model begins as a copy of the instruct model, also acting as both the expert (generating QA pairs from the context. e) and the teacher (producing token-distribution targets for the student). During training, the expert and teacher remain unchanged while the student learns by updating its LoRA adapter, gradually diverging from its initial state. Generally, the expert and teacher models can be chosen independently, except that logit-based distillation requires shared vocabulary between the student and teacher.
We systematically evaluate how changing the expert or teacher model affects PD and SFT performance (Table 3). SFT has no teacher, only the expert that generates larget answers, equivalent to the standard distillation approach via closed APIs. For
self-distillation, that is, using the same model to generate both data and logits (expert = teacher = 8B), yields superior knowledge injection compared using a larger Hlama-3-70B-Instrute or Qwen2.5-72BBInsttrrre 2wen2.5-14BB-nsttree
as either the expert or teacher. Similarly, for self-distillation again proves more effective than relying on data/logits from 72B model in the same family, likely because style mismatches overshadow any benefits of the more powerful model. An exception ariscs for the smaller model, Qwen2.5-3B, which struggles to generate high-quality training data. Using a larger expert with the 3B model as its own teacher improves results here, but even seff-distillation still outperforms standard SFT. Because Qwen2.5-3B has a different vocabulary from its larger siblings, logit-based distillation from the bigger models not directly possible.
Our findings suggest that mismatches in answering style often outweigh the benefits of more powerful expert teacher in knowledge injection. Especially under parameter-officient fino-tuning, larger teacher may inadvertently push the student to mimic the teacher's style rather than absorb new facts. Hence, self-distillation, where student = teacher = expert, often leads to better knowledge retention.
4.5 Ablations and Scaling Behavior
In this section, we present further experiments to analyze prompt distillation with Llama-3-8B-Instruuc. We provide detailed results from these ablations in Appendices Additionally, in Appendix J we explore reverse KL divergence as loss function, and in Appendix we analyze why using larger expert models does not always enhance performance.
Temperature. In Table we show the importance of using higher sampling temperatures for training data generation (7) and during distillation (T). Reducing the value of either temperature leads to a dogradation the PD performance. This highlights the importance of maximizing data coverage, actively learning to avoid incorrect answers, and leveraging the flatter probability distribution produced by the higher temperature,
10

---

Published in Transactions on Machine Learning Research (08/2025)
Correctness Training Questions
Correctness Training Questions
95
9x
Correctness (%)
90
Correctness (%)
90
3x
85
80
80
75
Prompt Distillation
Bonito
Supervised Fine Tuning
10
100
200
5
10
20
50
100
Training questions
Training questions
Figure 3: Average correctness (%) of PD and SFT 88. the number of questions per test question.
Figure 4: Answer correctness (%) of PD with Bonito vs. Llama-3-8B-Insrruct for question generation.
Correctness vs Wall-Clock Time
Correctness vs Rank
100
90
95
80
90
Correctness
Correctness
70
60
80
50
Prompt Distillation
Prompt Distillation
Supervised Fine-Tuning
Supervised Fine-Tuning
75
900
1800 2700 3600 4500 5400 6300 7200 8100 9000 9900
16
32
64
128
256
512
Training Time (s)
Rank
Figure Answer correctness (%) of PD and SFT us. training time (seconds).
Figure 6: Answer correctness (%) of PD and SFT
LoRA adapter rank.
which encourages the model to pay increased attention the probabilities of alternative tokens. In contrast, SFT uses the expert answers directly as the ground truth. While higher temperatures improve data coverage, they also introduce lower-quality answers, ultimately degrading SFT performance.
Number of Training Questions. In Figure 3 and Appendix F we study the impact of the number of training questions on the performance of PD and SFT on Jlama-3-SDB-Instruect These results highlight. that PD substantially more data-efficient than SFT, with the efficiency gap widening as the number of training questions increases. To match the performance of PD with 20 training questions, SFT needs an order of magnitude more training data. In our experiments, SFT did not reach the asymptotic performance of PD.
Question-Generating Model. In Figure 4 analyze the role of the training question generating model by comparing few-shot prompting of Llama-3-8B-Instruu to our reproduction of Bonito (Nayak et al. 2024). This model has been fine-tuned to generate zero-shot questions given the document. We hypothesize that Bonito's slight performance advantage at larger training question counts stems from improved fact coverage. However, the benefit is minor. For smaller data budgets, few-shot prompting is sufficient.
Wall-Clock Time. In the simplest PD implementation, each gradient step requires two forward passes: one without the LoRA adapter (to retrieve teacher logits) and one with it (for the student).. In contrast, SFT
11

---

Published in Transactions on Machine Learning Research (08/2025)
Table 5: The average answer correctness (%) on the question answering task with Llama-3-8h -B--truuutttt and the Tilu 3 dataset used to prevent catastrophic forgetting. The uncortainty is two standard errors.
Method
Amazon
New
Wiki
NYT
Reddit
Prompt Distillation
86.1
#0.2
94.4
#0.3
93.6
#0.6
79.5
#1.4
Prompt Distillation + Tülu 3
84.9 +0.4
94.5
+1.1
92.9
+0.3
79.0 +0.3
Prompt Distillation XL + Tülu 3
89.4 +0.5
96.2
#0.4
95.9
+0.4
80.3
+0.8
Alamm-3-8B-Insuruue + RAG
86.3
95.6
96.3
78.6
Table 6: The average answer correctness (%) of prompt distillation and supervised fine-tuning on MMLU-
Pro. The mean computed over 6 subjects. PD suffers less from lorgetting than SFT.
Method
Engineering
History
Law
Math
Physics
Psychology
Mean
Prompt Distillation
27.8
36.7
23.4
33.0
29.2
53.9
34.0
Prompt Distillation + Tülu
31.1
41.0
25.8
35.1
32.3
57.8
37.2
Prompt Distillation XL + Tülu 3
31.6
41.2
26.8
35.5
34.3
57.9
37.9
Supervised Fine-Tuning
28.4
32.6
22.6
30.3
26.9
50.7
31.9
Alama-3--B-Innstee Llama-3-8B
31.3
42.3
26.5
36.1
34.4
59.4
38.3
25.5
36.2
19.6
30.4
31.4
53.3
32.7
does not require a teacher pass. To analyze the impact on wall-clock time, we trained Llama-3-8B-Instruce saving model snapshots every 900 seconds and evaluating performance. To isolate efficiency differences rather than asymptotic performance, we used training questions for PD and 200 for SFT, leading to similar final performance (see Fig. 3). The results (Fig. 5 show that PD's higher data efficiency compensates for its additional forward pass, making it more time-efficient than SFT. Note that PD with self-distillation be further optimized by pre-computing and storing teacher logits during answer generation.
LoRA Rank. We used large LoRA adapter sizes in most experiments to ensure that model capacity was not a limiting factor. However, in real-world applications, memory constraints may require smaller adapters. To assess this impact, we varied the adapter rank for both PD and SFT while keeping LoRA alpha equal to twice the rank. The results (Fig. fig.10 6 show that both methods scale predictably, with performance increasing monotonically as the number of learnable parameters increases. PD consistently outperforms SFT at every rank, demonstrating that it is also more parameter-efficient approach to knowledge injection.
4.6 Prompt Distillation with Regularization
Finally, we investigate the efficacy of PD training in mitigating catastrophic forgetting when augmented with regularization. We use the Tühu 3 SFT Dataset (Lambert et al. 2024) regularization mechanism, with Mlama-3-8B-Instruce as the student model. Mini-batches contain equal proportion (50:50) of samples from the PD and Tülu 3 datasets. We applied the standard KL divergence loss function for items from the PD dataset (Eq. 1 In contrast, the regularization loss (Eq. 2 was utilized for Tülu dataset instances where no additional context e was present. This loss trains the student and teacher outputs to match.
We compared the trained model standard PD and SFT across our evaluation datasets, with MMLU-Pro (Wang al. 2024b) serving proxy metric for general model performance. The empirical results (Tables and 6) show that incorporating the Tülu 3 dataset slightly reduces performance in knowledge injection tasks but significantly improves resistance to catastrophic forgetting on MMLU-Pro. This approach narrows the gap to Llama-3--BB-Inttuuch to approximately one percentage point, demonstrating strong retention of general capabilities. Standard PD is still superior to the base Llama-3-8B model. In contrast, SFT on the low-temperature PD dataset significantly dograded instruction tuning benefits on MMLU-Pro, causing performance to drop below the base model's. These results suggest that using soft labels in PD mitigates performance degradation compared to SFT with one-hot targets.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Table 7: The average answer correctness (%) on HotpotQA across all methods and base models, in the closed-book (top) and RAG (bottom) settings. Uncertainty is two standard errors of the mean.
Method
Llama-3-8B- Instruct
Qwen2.G-14B- Instruct
Qwen2.5-3B- Instruct
PD (30 Qs)
70.0
+0.8
71.1
22.5
57.9
10.9
PD (100/50 Qs)
73.7 #0.1
73.4
#0.2
59.1
#2.5
SFT (30 Qs)
63.3 12.9
66.8
#0.4
56.4
#0.9
SFT (100/50 Qs)
68.9 +2.7
68.8
+2.2
51.8
+2.9
Base Model
53.0
43.5
28.8
PD (30 Qs) + RAG
81.7 +
86.0
+1.4
76.4
+1.9
PD (100/50 Qs) + RAG
82.1 +1.7
87.0
+0.3
76.9
114
SFT (30 Qs) + RAG
74.8 +1.8
80.2
14.1
75.1
11.0
SFT (100/50 Qs) + RAG
79.6 +13
78.5
112.1
75.2
+02
Base Model + RAG
79.5
83.5
73.5
To improve performance across ovaluation datasets and MMLU-Pro, we introduced the following three modifications to PD to form PD XL: (1) increasing the number of training questions from 30 to 200 per test question, (2) replacing Baama-3-B--Instee (3) limiting training to single epoch. Then, we combined PD XL with regularization based on the Tülu 3 dataset, leading the Prompt Distillation XL + Tülu 3 variant, which evaluated with Llama-3-8B- Instruct as the base model. This optimized approach reduces the performance gap to less than 0.5 percentage points relative to 1lama-3-SB3-nstreme
with our reproduction of Bonito for question generation, and
on MMLU-Pro while exceeding the OpenAl Embedddings-base basedd RAG baseline on knowledge injection tasks. Incorporating preference fine-tuning and RLVR, key post-SFT steps in the Tülu 3 pipeline [Lambert et al. 2024], could potentially further close the gap.
4.7 Experiments on HotpotQA
While our evaluations Squadshifts demonstrate PD's capacity to learn and retain factual knowledge, many rcal-world applications require modcls to reason using this incorporated knowledge to answer complex questions. To assess this capability, evaluate our method Hotpoi (Yang el al. 2018), a multi-hop QA benchmark designed to test reasoning. To answer the test questions successfully, the model has to synthesize information from multiple supporting documents to arrive at the correct answer.
We use the first 1,000 questions from the validation set of the Hotpot QA distractor setting for our experi- ments. Each test question typically associated with ten context paragraphs: two containing the necessary supporting facts and cight being distractors. Our objective is to inject the knowledge from these ten para- graphs into the student's 's LoRA adapter. This allows the student model to perform multi- hop reasoning to answer the test questions in closed-book setting. We generate single-hop training questions based on each provided paragraph separately to ensure the model can learn the factual content from each of the context paragraphs. The test questions, on the other hand, require multi-hop reasoning. Thus, we are testing PD in an out-of-distribution setting where test questions are more complex than the training questions. However, while our current question generation pipeline focuses on single-document contexts for simplicity and specifically lest the generalization from standard QA to multi-hop reasoning, there is no inherent limita- tion to PD that would prevent it from being trained directly for multi-hop reasoning. This could involve developing a pipeline to select multiple related documents and generate training questions that explicitly require synthesizing information, similar to how the HotpotQA dataset has been constructed. However, such an extension is beyond the scope of the current work, where our focus is on evaluating PD for general knowledge injection.
We follow the self-distillation principle from the Squadshifts experiments. An expert model generates the QA pairs, teacher generates the logits, and student equipped with a LoRA adapter is trained. All three
13

---

Published in Transactions on Machine Learning Research (08/2025)
use the same model. For PD, we sample the expert-generated answers at a temperature of 1.5. We use SFT as baseline and generate the answers at a temperature of 0.25, also aligning with Squadshifts.
We evaluate two main training data configurations for HotpotQA. In the first setting, we generated 30 training questions from each of the ten context paragraphs associated with a HotpotQA test question. We train both JJamm-3--8-Instrucct and the Qwen models for five epochs. Then, to explore the impact of greater data density per paragraph, which might benefit the multi-hop task, we increased the number of training questions to 100 per paragraph for -ans--et--Instruct and 50 per paragraph for the Qwen-based models. We used smaller number for Qwen models, as their outputs were generally more verbose. In this configuration, the -lama-3-8B-Instuue for three. These settings were chosen based on preliminary experiments to optimize performance within our computational budget.
model was trained for two epochs, and the Qwen models were trained
We evaluate performance in two scttings: a closod-book scenario without RAG and using RAG with OpenAI Embeddings (specifically, lext-embeedding---mmalll
For RAG, we retrieved the top k 10 most relevant paragraphs from the pool of all paragraphs associated with our 1,000 test questions. The value of k was chosen to align with the standard HotpotQA setup, where 10 context paragraphs are used in conjunction using the same Chain-of- Thought prompting methodology as used for Squadshifts. For completeness, results using substring match grading, following Mallen et al. 2022). are provided in Appendix N
with each question. Answer grading was performed by -lama-3--B-Instaee
The results are presented in Table 7 They demonstrate that Prompt Distillation consistently outperforms
both SFT and the base models across all tested architectures, data configurations, and evaluation settings on HotpotQA. Incrcasing the number of training questions per paragraph from 30 to 100/50 generally leads to improved performance for PD, which highlights its ability to utilize more extensive training data effectively, Our results suggest that PD not only effectively injects new knowledge (as shown with Squadshifts) but also enhances the model's ability to utilize this newly acquired knowledge in more complex, multi-step reasoning tasks. When combined with RAG, models trained with PD generally show superior performance, which indicates that, the model can better combine the injected and retrieved information for multi-hop reasoning. Our results suggest that PD enables deeper assimilation of knowledge rather than superficial recall, which allows PD to apply the knowledge in novel and more demanding scttings.
5 Conclusion
In this paper, we propose using prompt distillation (PD) to inject knowledge from free-form documents into LLMs and analyze factors impacting its performance. Our method uses delf-distillation, where a student model learns to replicate the answers of a teacher model with access to the target knowledge in its prompt. We show that PD outperforms traditional supervised finc-tuning (SFT) across model sizes and familios, achieving higher accuracy with less data while being more efficient in terms of wall-clock time and paramelers. Unlike SFT, PD does not depend on larger teacher models, provided the base model is sufficiently capable, In knowledge injection tasks from Squadshifts, PD achieves closed-book book performance competitive with RAG when the number of training questions and the LoRA rank are sufficient. To the best of our knowledge, this has not been achieved with SFT. Furthermore, on the complex multi-hop reasoning benchmark Hotpot PD substantially improves the base model's performance and, notably, can enhance the performance of RAG systems when used in conjunction.
We believe storing knowledge within model weights offers a scalable alternative to RAG systems, which rely on long prompts and external databases, making them prone to retrieval errors. Effective knowledge internalization requires integrating new information with prior knowledge. PD can facilitate this more naturally than structured knowledge bases, which require ongoing maintenance. PD could be especially useful for domains like corporate knowledge management, where information is largely unstructured. One of the main challenges in this study was the absence of standardized benchmarks for unsupervised knowledge injection. We addressed this by modifying Squadshifts data to ensure that questions remained understandable without their original context, making comparing RAG and fine-tuning more fair. This modification improved the performance of RAG-based methods through improved retrieval quality. The modified questions have been included in the supplementary material. For evaluation, we primarily relied L.LM-as-a-judge hods
14

---

Published in Transactions on Machine Learning Research (08/2025)
using models from different families (Llama & Qwen), verifying that grading outputs were consistent and rankings remained unaffected. Furthermore, we used substring match grading, which generally yielded similar conclusions despite variances in absolute scores. We also manually verified a subset of the LLM grades to ensure reliability. Finally, if the base model is too small, using a more powerful model as the expert can improve performance.
Our knowledge injection technique could aid the development of agentic LLMs, potentially enabling self- improving models that distill new insights directly into their weights. Future research could explore hybrid approaches combining PD with RAG and investigate how distractor documents impact model robustness. A particularly promising future work direction is optimizing and diversifying training data generation strategies for Prompt Distillation. For instance, optimizing question generation to maximize fact coverage or developing training questions could further improve PD in factual recall and multi-hop
methods for reasoning, respectively. More generally, investigating how tailored synthetic data can lead to even greater performance for PD in wide range of downstream tasks could further extend PD's capabilities.
Acknowledgments
This work was supported by the Research Council of Finland (Flagship programme: Finnish Center for Artificial Intelligence FCAI, and grants 352986, 358246) and EU (H2020 grant 101016775 and NextGener- ationEU). We thank Joni Pajarinen for helpful discussions and Minttu Alakuijala for contributions to the codebase. We thank CSC IT Center for Science in Finland for providing access to the LUMI supercomputer, and the Aalto Science-IT project for additional computational resources.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tgge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXit:2303.0877 08774, 2023.
Anthropic. Claude 3.5 sonnet model card addendum, 2024. URL https: http://wwwwcdn.aaatrrppip. anthropic com/ 12-02.
Ge66c398488411132277888877786844 Card_ Claude Addendum. pdf Accessod: 2024-
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant a laboratory for alignment. arXiv preprint ar XrX:::2112.000 008611, 2021.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pp.
PMLR, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurol information processing systems, 33:1877 1901, 2020.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Vosse Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang, Quantifying memorization across neural language models, arXiv preprint N...:22220076444 2022.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models retrieval- augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17764- 2024.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. arXiv preprinl XrXi::335.. 14788, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzaloz, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. URL vicuna. lamiaa.mmsssorrr.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVE international conference computer vision, pp. 4794-4802, 2019.
Eunbi Choi, Yongrae Jo, Joel Jang, and Minjoon Seo. Prompt injection: Parameterization of fixed inputs. arXiv preprint ar Xic:22006. 1 11499, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint arXiv:2407.2178. 2024.
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms. arXiv preprinl 151112305.1577,, 2023.
Aman Gupta, Amup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al. Rag vs fine-tuning: Pipelines, tradeoffs, and a case study on agriculture. arXiv preprint arXiv:2401.08400 2024.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kylc Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and lasks. arXiv preprinl arXiv:2004.10964, 2020.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference machine learning, pp. 3929 3938. PMLR, 2020.
Junxian He, Jiatao Gu, Jiajun Shen, and Marc' Aurelio Ranzato. Revisiting self-training for neural sequence generation. arxiv preprinl Xrxio:1909. 13788, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXie preprint arXiv:1503.0955 02531, 2015.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Jie Huang, Wei Ping, Peng Xu, Mohammad Shoeybi, Kevin Chen-Chuan Chang, and Bryan Catanzaro. Raven: In-context learning with retrieval augmented encoder-decoder language models. arXiv preprint arXiv:2308.0000 2023.
Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering, arXiv preprint arXio:2007.0198: 01282, 2020.
Gautier Izacard, Patrick Lewis, Maria. Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1-44 43, 2023.
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. arXiv preprint
2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 -3526, 2017.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Facze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tạ" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXie:2411.
2024.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge- intensive ulp lasks. Advances in Neural Information Processing Systems, 33:9459 9474, 2020.
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, el al. Ra-dit: Relrieval-augmented dual instruction luning. arXiv preprint Xrx:::2310. 0.3552 2023.
Kai Liu, Ze Chen, Zhihang Fu, Rongxin Jiang, Fan Zhou, Yaowu Chen, Yue Wu, and Jieping Ye. Ed- ucating 1mss like human students: Structure-aware injection of domain knowledge. arXiv preprint arXiv:240.11672, 2024a.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Compulational Linguistics, 12:157 173, 2024b.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompling methods in natural language processing. ACM Computing Surveys, 55(9):1-33 35, 2023.
Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv preprint arXiv:2401. 10225, 2024c.
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2421-2425, 2024.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannanch Hajishirzi, and Daniel Khashabi. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXie preprint arXiv:2212. 10511, 1, 7, 2022.
Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, et al. Injecting new knowledge into large language models via supervised finc-tuning. arXiv preprint arXiv:2404.0000 00213, 2024.
John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift question answering models. In International conference machine learning, pp. 6905 6916. PMLR, 2020.
Scyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 34, pp. 5191-5199 2020.
Arindam Milra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, et al. Orea 2: Teaching small language models how to reason. arXiv preprint 2rXiv:2311.11044 2023.
Subhabrata Mukherjee and Ahmed Awadallah. Xtremedistil: Multi-stage distillation for massive multilingual models. Xrviv preprint ar arXiv:2004.05680 055686 2020.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXio preprint arXiv:2306.0211 2023.
Nihal V Nayak, Yiyang Nan, Avi Trost, and Stephen Bach. Learning to generate instruction tuning datasets for zero-shot task adaptation. arxio preprint wxXim:2402.18333 2024.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Oded Ovadia, Menachem Brief, Moshik Mishacli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in llss. arXiv preprint arXiv:231 059344, 2023.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4.
arXiv preprint 23344 03277,,077777 2023.
Siyuan Qi, Bangcheng Yang, Kailin Jiang, Xiaobo Wang, Jiaqi Li, Yifan Zhong, Yaodong Yang, and Zi- long Zheng. In-context editing: Learning knowledge from self-induced distributions. ar preprinl arXiv:2406. 111194 2024.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmenter language models. Transactions of the Association for Computa- tional Linguistics, 11:1316-1331, 2023.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beanlien Mike Gatford, et al. Okapi at tree-3. Nist Special Publication Sp, 109:109, 1995.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured pre- diction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and slatislics, pp. 627 635. JMLR Workshop and Conference Proceedings, 2011.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint
108, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprinl ar Xrxvv:1 10. 0800827, 2021.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. 31210-31227. PMI.R, 2023a.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Replug: Retrioval-angmenned black-box language models. arXiv preprint orXiv:2501.12552, 2023b.
Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context. Xr preprint arXiv:2209.155 151899 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-folloving llama model, 2023. URL https: alpaca.
//ithub. wwtp:///ttttttaaaatttaaaaee
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Boxin Wang, Wei Ping, Lawrence McAfee, Peng Xu, Bo Li, Mohammad Shoeybi, and Bryan Catanzaro. Instructretro: Instruction tuning post retrieval-augmente augmented pretraining. arXiv preprint arXiv:2310.07713. 2023.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 83:5776-5748 2020.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Xiangfeng Wang, Zaiyi Chen, Zheyong Xie, Tong Xu, Yongyi He, and Enhong Chen. In-context former: Lightning-fast compressing context for large language model. arviv preprint arXiv:206. 13618, 2024a.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Tight Conference on Neural Information Processing Systems Datosets and Benchmarks Track, 2024b.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.122 2023a.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: An open-source chat model with parameter- efficient tuning on self-chat data. arXiv preprint or Xix:2304.01190 2023b.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving rotrieval-augmenter Ims with context compression and selective augmentation. In The Twelfth International Conference on Learning Represen- tations, 2024a.
Ran Xu, Wenqi Shi, Yue Yu, Yuchen Zhuang, Yanqiao Zhu, May D) Wang, Joyce C Ho, Chao Zhang, and Carl Yang. Bmretriever: Tuning large language models as better biomedical text retrievers. XrXiv preprint arXiv:2404. 2024b.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1111 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint (XX:::1000000000
2018.
Qichen Ye, Junling Liu, Dading Chong, Poilin Zhou, Yining Hua, and Andrew Liu. Qilin-med: Multi-stage kuowledge injection advanced medical large language model. arXie preprinl arXiv:2310.0m 09089, 2023.
Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E Gonzalez. Raft: Adapting language model to domain specific rag. arXiv preprint ar Xrx:22403 10131, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Lin, Zhuohan Lá, Dacheng Li, Eric Xing, al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Newral Information Processing Systems, 36:46595 46623, 2023.
Junyao Zhou, Ruiqing Du, Yushan Tan, Jintao Yang, Zonghao Yang, Wei Luo, Zhunchen Luo, Xian Zhou, and Wenpeng Hu. Context compression and extraction: Efficiency inference large language models. International Conference on Intelligent Computing, pp. 221-232, Springer, 2024.
A Reproducibility
The codebase for implementing prompt distillation, the evaluation questions, and instructions for replication of the main results can be found in the following GitHub repository: htttp://ggthuu..comkkkllaa prompt-distillen distillation
B Related Work: Retrieval-Augmenn
Language Models
Retrieval- Retrieval-Augmenteed Language Models integrate retrieval directly into the language model architecture Guu
et al. 2020 Borgeaud et al. 2022 Khandelwal al. 2019. Izacard et al. 2023 Huang et al., (023). Large
autoregressive transformers can be retrofitted with retrieval capabilities during a fine-tuning or a continual
pre-training phase (Lin et al. 2023 Liu et. al. 2024c Wang et al. 2023), they can be converted into explicit
19

---

Published in Transactions on Machine Learning Research (08/2025)
The LLM can also be combined with retriever Methods to compress the retrieved context. for more effective inference (Xu et al. 2024a Zhou et al. 2024 Wang et al. 2024a Chevalier et al. 2023) are related to our work, as we aim to compress context directly into model weights through prompt distillation, reducing the need for complex retrieval pipelines and making inference cheaper.
retrievers for RAG Ma et al. 2024 Xu et al. 2024b).
without fine-tuning (Shi et al., 2023b Ram et al. 2023).
C Related Work: More Detailed Review of Context Distillation
In prior work, context distillation has been used for in-context learning and qualitatively modifying LLM behavior. The idca of distilling context into model weights was used by Askell et al.] (2021) to improve LLM alignment with human values to produce polite, helpful, and accurate answers. They distill a prompt that contains fourteen human-assistant conversations in which the assistant always follows the desired policy. Snell et al. (022)) demonstrated the distillation of more detailed task instructions, in-context examples, and stppbyy-tepp reasoning. Choi et al. (0022) used the distillation idea to distill short instruction prompts that, for example, define a persona participating in a conversation or contain task instructions. They additionally trained a model to generate input data for the distillation process, using examples from the considered benchmarks as training data.
Qi al. (2024) proposed using distillation approach edit knowledge retained in the LLM weights. The factual knowledge is assumed to exist as triplets (entity, relation, object). The triplets converted into short statements by large proprietary LLMs, which are used additional context for the teacher's prompt. Compared to that paper, we neither assume the existence of new knowledge in the triplet form nor data generation by larger LLMs. Instead, we learn new facts from raw documents without any supervision.
20

---

Published in Transactions on Machine Learning Research (08/2025)
D Hyperparameters
Table 8: Hyperparameters used in our experiments. We either performed a grid search determine the optimal values or selected the largest feasible values that fit within our computational budget and GPU memory.
Hyperparameter
Value
Temperature for question generation (Llama & Qwen)
1.5
Temperature for question generation (Bonito)
1.25
Temperature for teacher answer generation (PD)
1.5
Temperature for teacher answer generation (SFT)
0.25
Maximum teacher answer length, PD datasct (tokens)
512
Maximum total prompt length, PD dataset (Lokens)
768
Maximum total prompt length, Tilu 3 dataset (tokens)
1024
Temperature for sampling evaluation answers Maximum answer length, evaluation Number of documents to be retrieved Learning rate Temperature for KL divergence loss Batch size
0.25
500
1e-5
2.0
Batch size (Low-rank experiments & PD XL & Qwen2.5-14B 1----rutttttt
32
Gradient accumulation steps Adam weight decay
0.1
Maximum gradient norm
1.0
Precision
BF16
GPU
AMD InstinetTM MI250X
LoRA rank (3B, 8B)
1024
LoRA rank (14B)
512
LoRA type
Full
LoRA alpha
rank
Learning rate warmup duration (Llama-3)
2 1 epoch
Learning rate warmup duration (Qwen2.5)
100 steps
Learning rate warmup lype
Linear
Training duration (Llama-3)
10 epochs
Training duration (Qwen2.5)
5 epochs
Training duration (PD XL)
1 epoch
Training duration (UFT)
3 epochs
Learning rate for UFT
5e-5
21

---

Published in Transactions on Machine Learning Research (08/2025)
E
Results Obtained with RAG
The average answer correclness (%) of prompt distillation and baseline methods in the RAG-based setting with Llama-3-8B-Issruuct.. Retrieval was performed using BM25 and OpenAl Embeddings with cosine similarity. The uncertainty two standard errors of the mean.
Table 9:
Method
RAG
Amazon
New Wiki
NYT
Reddit
Prompt Distillation
BM25
87.0
#0.4
96.4 #0.6
96.7
#0.1
70.5
#2.7
Prompt Distillation
emb
88.5 +0.3
96.7
+0.2
96.9
+0.2
77.8
+1.6
Supervised Fine-Tuning
BM25
85.4 +0.1
95.6
+0.5
96.0
+0.3
71.5
+2.5
Supervised Fine-Tuning
emb
87.3 #0.6
95.8
10.2
96.2
10.3
75.2
42.9
Unsupervised Fine- Taning
BM25
67.7 #1.1
87.5 #1.8
85.8
#0.6
54.1
#2.7
Unsupervised Fine-Tuning
emb
68.4 +1.3
87.0 +1.8
87.1
+1.8
57.7
+1.7
SFT w/ Distractors
BM25
84.0 #0.6
95.6 #0.4
94.3
10.5
77.9
#0.4
SFT w/ Distractors
emb
87.0 +0.7
95.1 +0.4
94.2 10.1
82.4 +0.8
Baitaa-----Innnstte 1lama-3-8B-Insse
BM25
82.4
94.4
94.8
69.2
emb
86.3
95.6
96.3
78.6
Llama-3-88
Oracle
94.7
98.4
98.1
84.3
Table 9 presents the results of prompt distillation (PD) and baseline methods in rotrieval-angmennedl generation (RAG) setting. We compare two retrieval strategies: BM25 and OpenAl Embeddings (denoted as 'emb"). PD outperforms standard supervised fine-tuning (SFT) across all datasets and achieves the highest accuracy with OpenAI Emboddings-bant the RAG setting on the Reddit dataset. The table also includes results for unsupervised fine-tuning with RAG, which underperforms compared to other methods. As an upper bound, we report results using an 'Oracle' retrieval method, where the correct document is always retrieved. This highlights the impact of retrieval quality, with all methods falling short of Oracle's performance due to imperfect retrieval.
retricval. Adding distractors to SFT improves its robustness in
22

---

Published in Transactions on Machine Learning Research (08/2025)
F
Ablation: Impact of the Number of Training Questions
Table 10: The average answer correctness (%) of prompt distillation with a varying number of training questions generated by Bonito per test question on the New York Times dataset with Llama-3-8B-Instruce Retrieval was performed using BM25 and OpenAT Embeddings with cosine similarity. The uncertainty is two standard errors of the mean.
Method
5
10
20
30
50
100
200
PD
83.0
+0.0
89.0
10.1
93.0
10.6
93.7
+0.2
95.5
#0.6
96.5
#0.1
96.8
10.7
SFT
72.4
+1.
78.6
=1.3
84.8
#0.6
87.0
#0.8
89.1
#0.7
91.3
#0.4
93.3
#0.1
PD + BM25
95.2
#0.4
95.3
#0.2
96.2
=0.2
96.2
#0.3
96.6
#0.2
96.6
#0.1
96.2
#0.1
SFT + BM25
94.1
+0.6
94.4
+1.1
94.9
+0.2
95.2
+0.5
95.3
+0.3
95.7
#0.1
95.5 +0.2
PD + emb
95.8
+0.5
96.2
10.2
97.0
+0.5
96.9
#0.4
96.9
#0.2
97.0
#0.3
96.9
10.7
SFT emb
94.9
#0.3
95.0
#0.4
95.4
#0.4
96.3
#0.1
95.4
#0.4
96.1
#0.5
96.6
#0.5
Table 10 compares the sensitivity of prompt distillation and supervised fine-tuning with token loss to the number of questions per document in the training data with Llama-3-9B-Instrucc, We sampled the questions using Bonito (Nayak al. (024)) in this experiment. We found Bonito capable of generating competitive questions for the New York Times dataset. Inereasing the number of questions provides statistically significant benefit, particularly in the closed-book setting. For instance, the performance NYT increases from 83.0% with five questions to 93.7% with 30 questions.
We observe that prompt distillation much more samplc-effciintt than supervised finc-tuning with token loss, reaching similar performance with much fewer samples. We also see that the performance of supervised fine-tuning saturates on a slightly lower level than prompt distillation the RAG setting, showcasing that prompt distillation superior convergence. Finally, note that the baseline performance for Llama-3-8B- Instruct on NYT was 38.2%, showing that even small number of questions gives significant benefits in terms of the ness percentage.
23

---

Published in Transactions on Machine Learning Research (08/2025)
G Ablation: Effect of Question-Generating Model
Table 11: The average answer correctness (%) of prompt distillation with Bonito and Llama-3-8B-Instruct used for gencrating the training question on the New York Times dataset, with Llama-3-8B-Intruuc the trained model. The upper part of the table corresponds to the closed book results, whereas the lower half consists of the results in the RAG setting, Retrieval was performed using OpenAl Embeddings with cosine similarity. The uncertainty is two standard errors of the mean.
Method
5
10
20
30
50
100
Bonito
83.0
=0.0
89.0
#0.1
93.0
#0.6
98.7
=0.2
95.5
#0.6
96.5
#0.1
Haama-3-8B-Instruet
84.6
+0.7
89.8
+0.5
92.3
+0.2
93.6
+0.6
94.9
+0.1
96.0
+0.3
Bonito emb
95.8
+0.5
96.2
+0.2
97.0
+0.5
96.9
#0.4
96.9
+0.2
97.0
+0.3
-lama-3-8B-Instee Instruct + emb
94.9
#0.3
95.4
+0.3
96.2
+0.3
96.9
#0.2
97.3
#0.2
97.1
#0.3
Table 11 compares the performance of prompt distillation (PD) when using Bonito versus Llama-3-8B- Instruct to generate training questions the New York Times dataset. We evaluate correctness across different numbers of training questions beddings. The results do not indicate any significant differences between the methods. As the number of training questions increases, Bonito exhibits a marginal advantage in closed-book settings. However, with RAG using OpenAI Embeddings, performance converges at similar level for both pnestion-generatie proaches, suggesting that retrieval mitigates differences in training question diversity. These findings suggest that while fino-tuned question-generaaiion model may provide benefits at scale, few-shot prompting of general-purposs model is viable alternative for PD.
test question, with and without retrieval using OpenAl Em-
ap-
24

---

Published in Transactions on Machine Learning Research (08/2025)
H Ablation: Training Time Efficiency
Table 12:
The average answer correctness (%) of prompt distillation and supervised fine-tuning as function
of training time (in seconds) on the New York Times dataset with Llama-3-SB-Instrucc The uncertainty is two standard errors of the mean.
Training Time (s)
PD
SFT
900
58.5 +0.7
41.0
+0.2
1800
76.8 +0.5
56.9 #1.1
2700
84.7 =0.3
70.1 #1.0
3600
87.8 +0.5
78.1 +0.3
4500
89.3 +0.3
82.6 +0.6
5400
91.1 10.6
85.5 #1.0
6300
91.0 #0.7
85.4 #0.3
7200
91.8 +0.3
88.5 +0.7
8100
92.1 10.1
89.4 10.4
9000
92.4 +0.2
90.8 +1.2
9900
93.0 10.5
91.3 #0.7
Table 12 presents the relationship between training time and performance for Prompt Distillation (PD) and Supervised Fine- Tuning (SFT) on the New York Times dataset using Llama-3-8B-Instruce To ensure comparable asymptotic performance, The results show that PD consistently outperforms SFT at every time interval, achieving higher correctness scores with fewer training steps. For instance, PD reaches 90% correctness in approximately 5,000 seconds, whereas SFT takes nearly twice as long to approach similar performance. These findings highlight that PD's superior sample efficiency outweighs the additional computational cost of two forward passes- one for the student and another for the teacher. Due to storage constraints and slow file systems, we did not implement optimizations such pre-computed logits.
used 30 training questions per test question for PD and 200 for SFT.
25

---

Published in Transactions on Machine Learning Research (08/2025)
Ablation: Impact of LoRA Rank on Performance
Table 13: The average answer correctness (%) of prompt distillation and supervised fine-tuning as function of LoRA rank on the New York Times dataset. The uncertainty two standard errors of the mean.
Method
16
64
128
256
512
SFT
81.0 +1.0
83.8
#0.6
83.8
+0.7
85.3
+0.1
85.9
#0.1
86.9
#0.1
PD
90.9 +0.0
91.5
+0.0
92.9
+0.0
93.7
+0.0
93.7
+0.0
94.7
+0.0
Table 13 presents the average answer correctness for prompt distillation (PD) and supervised fine-tuning (SFT) as a function of LoRA adapter rank on the New York Times dataset using Llama-3-8B-Instruuc.. We observe a clear trend where increasing the LoRA rank improves performance for both methods, with PD consistently outperforming SFT at all ranks. Even at the lowest rank (16), PD surpassos SFT at its highest, demonstrating PD's superior parameter efficiency in knowledge injection lasks.
26

---

Published in Transactions on Machine Learning Research (08/2025)
J Ablation: Reverse KL Divergence
The average answer correctness (%) of prompt distillation with different KT. divergences and temperatures with Llama-3-8B-Instrudd. The top row the standard prompt distillation configuration. The uncertainty two standard errors of the mean.
Table 14:
KL
T
T
Amazon
New Wiki
NYT
Reddit
Forward
2
1.5
86.1
+0.2
94.4
+0.3
93.6
+0.6
79.5
#1.4
Reverse
1
0.25
75.5
+0.8
88.4
+0.5
84.4
#1.0
70.2
#1.3
Reverse
1
1.5
77.3
=0.7
89.8
#0.2
86.7
#0.4
71.4
=2.2
Reverse
2
0.25
82.5
+0.8
93.2
+0.2
90.9
+0.4
76.3
+0.1
Reverse
2
1.5
84.1
#0.5
93.4
+0.7
92.1
+0.3
76.0
#0.6
Table 14 compares the performance of prompt distillation using forward KL divergence (standard con- figuration) versus reverse KT. divergence at different temperatures. Forward KT. consistently outperforms reverse KL across all datasets, demonstrating its superior effectiveness in knowledge injection. One intu- itive reason is that forward KL divergence (KL(teacher || student)) is mode covering, heavily penalizing the student for failing to capture the teacher's most probable tokens. By contrast, reverse KT. divergence (KL(student || teacher)) is mode seeking and does not penalize ignoring parts of the teacher's distribution as strongly. This disparity becomes especially problematic when the distillation temperature is lowered from two one because the teacher's distribution becomes sharper (i.e., higher peaks on certain tokens). Under forward KI, the student is forced learn these peaks more accurately (see Table 4 the drop in performance is smaller. Under reverse KL, the student more readily collapses onto few modes, struggling to match the teacher's distribution whole and leading to significant drop in performance.
27

---

Published in Transactions on Machine Learning Research (08/2025)
K
Ablation: Counter-Intuitive Scaling Effects with Bigger Experts
The average answer correctness (%) of prompt distillation on the New York Times dataset when training on different quintiles of answers generated by D0ven2.5-72B-Insttuue The training data divided into five disjoint subsets based on either the initial Kullback-Leibler divergence (KLD) between the student and teacher or the teacher's entropy. The uncertainty two standard errors of the mean.
Table 15:
Type
Quintile
Amazon
New Wiki
NYT
Reddit
Entropy
67.0
#1.2
75.0
10.5
71.1
10.3
66.9
#0.2
Entropy
2
72.0
+0.7
85.3
#0.2
79.4
#0.6
69.6
#1.1
Entropy
3
73.2
+0.4
84.5
+0.1
81.8
+1.0
65.9
+2.3
Entropy
4
73.0
#0.2
82.1
#0.7
80.9
#0.2
65.5
#1.0
Entropy
69.1
#1.9
77.0
#0.2
76.4
#1.0
57.2
#1.0
KLD
1
63.8
+1.3
70.6
+1.5
66.8
#1.4
50.2
#1.2
KLD
2
71.3
#0.6
79.5
+13
77.6
#1.0
59.7
#0.4
KLD
72.8
+1.7
82.0
+0.8
82.1
+0.3
66.6 +0.5
KLD
73.6
+0.4
86.8
10.6
84.1
#0.4
67.9
10.8
KLD
69.5 #0.4
84.1 #1.3
79.8
#0.4
68.8
10.8
Table 16: Comparison of teacher entropy and initial KT. divergence between the teacher and student models for different data-generating models across datasets, with Llama-3-SB-Instruue the student and teacher. In each block, the first row represents the standard prompt distillation setting, self-distillation.
Expert
Amazon
New Wiki
NYT
Reddit
Mean
Teacher Entropy
Blama-3-88B-Istrnic
0.41
0.26
0.30
0.68
0.41
Owen2.5-72B-Instan
0.52
0.42
0.43
0.71
0.52
KL Divergence Tlama-3-8B-Instrun
1.70
1.42
1.56
1.29
1.49
Qwen2..577BBlnnstruee
1.31
0.92
1.19
0.83
1.06
Our empirical results Table 3 reveal an unexpected phenomenon: knowledge injection performance deteri- orates when larger, more capable model serves as the auswer-generating expert. This observation contrasts with the conventional expectation that distillation from more powerful models consistently improves the performance of smaller models. While similar counter-intuitive effects have been noted in computer vision Mirzadch et al. 2020) and general reasoning tasks with LLMs (Gudibande et al. 2023 our study extends These findings to the specific case of knowledge injection through
Choo & Hariharan
2019
2023 Mitra el al.
fine-tuning.
To explore potential factors underlying this phenomenon, we examine two key statistical propertics of the teacher model's outputs:
1. We hypothesize that high entropy in the teacher's output distribution, possibly due to task difficulty or stylistic divergence between the lata-generating expert and the teacher, may lead to weaker or noisier training signals. Conversely, very low entropy might indicate that the base model already possesses the relevant knowledge, making additional knowledge transfer redundant.
The initial KL divergence between the student and teacher distributions represents a rough upper bound on potential information gain during fine-tuning. low divergence may imply that the student already knows the information that the teacher's uncertain signal could limit learning.
28

---

Published in Transactions on Machine Learning Research (08/2025)
We conducted experiments utilizing (lama-3-8B-Insturee the base model and Qwen2.5-72B-Instteee Instruct the onswer-generating expert model. We partitioned the training corpus into quintiles based entropy and KL divergence and independently fine-tuned Llama-3-8B- -anst-----lnstruuttt on each subset. As shown in Table [5] the results indicate that a higher imitial KL divergence correlates with greater performance gains, although extremely large values can have adverse effects. Both extremes perform poorly for teacher entropy, and training on the second quintile our setup yiclds the best results on average.
Comparative analysis of whole datasets (Table 16 generated by Llama-3-8B Inssructt and Qwen2.5-722 Instruct reveals that when the training data is self-generated by the model (Llama), the teacher entropy is lower while the initial KL divergence is greater. This characteristic may help explain why Llama-3- 8B-Instruct demonstrates superior performance as an expert compared to Qwen2.5-7BB-Instruue (Table 3 However, we emphasize that these are correlations. Further study would be required to isolate and confirm the causal mechanisms, as there might be confounding factors.
29

---

Published in Transactions on Machine Learning Research (08/2025)
L Manual Answer Grading
We performed manual evaluation of the grading accuracy of Blama-3-8B-Inntruuct. We evaluated 1000 grades assigned by the grader across three models: our prompt distillation model, the SFT model, and the default. Ilamt-t--i--nnsrucct. This evaluation covered the RAG and closed-book settings for PD and SFT and the RAG setting for Llama-3-8B-Insttnn with over 90% of these clearly incorrect grades being false negatives, that is, instances where the grader marked correct responses incorrect. The similarity in the proportion of incorrect grades across all models indicates systematic bias favoring model over another. Given the large volume of evaluations, manual grading would have been infeasible. We found -aama-3-BB-Instuue to be good balance between speed and grading accuracy.
The proportion clearly incorrect grades was low, at 2.2%,
Table 17: Confusion matrix for grading performed by Llama-3-81 3B-Instruc for Token Loss and Prompt Distillation.
Token Loss
Correct Label TRUE Correct Label FALSE Total
Graded TRUE
310
311
Graded FALSE
5
84
89
Total
315
85
400
Prompt Distillation
Correct Label TRUE
Correct Label FALSE
Total
Graded TRUE
327
328
Graded FALSE
9
63
72
Total
336
64
400
30

---

Published in Transactions on Machine Learning Research (08/2025)
M Answer Grading with Sween.5--2B-Instrre
We re-graded the most critical experiments using 101.225-33B--nstrrue to verify the reliability of our grading approach. The results are presented in the tables included in this appendix. The re-evaluation with Qwen2.5- 32R-Instruct demonstrated that the overall ranking of methods remained consistent. across different grading models. While minor variations were observed, these differences were generally small, except for non-fine- tuned models without retrieval-angmeented generation (RAG). In this sctting, Swren2.5-32--Inttie stricter grading criteria than Llama-3-8B-Instrucll, with particularly pronounced gap when Llama-3-8B-- Instruct graded its own outputs. However, the non-fie-tuned instruct model consistently exhibited the lowest performance. Consequently, the greater variance in grading does not affect the core conclusions of our study: (1) PD is superior to SFT across different settings,
applied
(2) PD combined with RAG outperforms the instruct model with RAG, and (3) the PD XT. + Tilu 3 model surpasses the instruct model with RAG.
M.1 Base Model: -arma-3-8B-Instiie
Table 18: Average answer correctness (%) on question answering task in the closed-book (upper part) and RAG (lower part) scenarios with alama-3-8B-Instrune when the grading has been done with Qwen2.5- 32s-Inst The uncertainty two standard crrors of the mcan.
Method
Amazon
New Wiki
NYT
Reddit
Prompt Distillation
85.6
+0.3
92.9
+0.5
92.9
#0.3
80.3
#0.5
Supervised Fine-Tuning
75.3 +1.4
87.5
+0.7
86.4
+0.2
68.8
+1.2
Unsupervised Fine-Tuning
39.6 +2.7
58.3
#1.0
50.5
#1.1
29.1 #2.7
Base Model
19.7
51.1
30.4
14.5
Prompt Distillation + RAG
88.9
#0.4
96.0
#0.2
96.7
#0.7
80.7
11.3
SFT w/ Distractors RAG
87.2
#0.4
93.5
+0.2
93.9
#0.2
83.6
#0.4
Base Model + RAG
85.3
94.9
95.6
78.4
Table 19: Difference correctness depending on the grader (Qwen minus Llama) in percentage points.
Method
Amazon
New Wiki
NYT
Reddit
Prompt Distillation
-0.5
-1.5
-0.7
+0.8
Supervised Fine-Tuning
-0.6
-2.0
-1.1
-1.0
Unsupervised Vine-Tuning
+0.1
-4.8
-2.1
-1.8
Base Model
-2.4
-10.1
-7.8
-6.3
Prompt Distillation + RAG
+0.4
-0.8
-0.3
+2.9
SFT w/ Distractors + RAG
+0.2
-1.6
-0.4
+1.1
Base Model + RAG
-1.0
-0.7
-0.7
-0.2
31

---

Published in Transactions on Machine Learning Research (08/2025)
M.2 Base Model: Twen2.5-14B-Instruu
Table 20: Average answer correctness (%) on question answering task in the closed-book (upper part) and RAG (lower part) scenarios with Swen2.5-14B-Instruee when the grading has been done with Qwen2.5- Instruct The uncertainty is two standard errors of the mean.
Method
Amazon
New Wiki
NYT
Reddit
Prompt Distillation
87.3
10.3
94.2
+0.3
92.8
#0.1
81.4 =0.2
Supervised Fine- Tuning
79.8 #1.2
88.5
#1.1
87.0 #1.0
72.3 #0.8
Swen2.5-14B--nstoae
17.7
56.4
27.7
13.5
Prompt Distillation + RAG
90.4 +0.2
96.7
+0.5
96.9
+0.3
86.6 +0.7
SFT + RAG
89.7 =0.2
95.5
+0.4
96.5
+0.4
86.1
#0.1
Qwen225511BB-nsstrcc + RAG
89.1
96.7
95.0
83.9
Difference in correctness with Qwen2:5511B-Instrucct the base model depending on the grader Table 21:
(Qwen minus Llama) in percentage points.
Method
Amazon
New Wiki
NYT
Reddit
Prompt Distillation
+1.8
-0.7
+0.2
+3.88
Supervised Fine-Tumin
+1.6
-1.7
-0,2
+0.8
Base Model
+1.2
-5.5
-3.5
-3.2
Prompt Distillation + RAG
+2.0
-0.6
+0.6
+3.0
SET + RAG
+1.8
-1.0
+0.8
+3.0
Base Model + RAG
+2.0
-0.4
+0.2
+2.6
32

---

Published in Transactions on Machine Learning Research (08/2025)
M.3 Base Model: Qwen2.5-3B-Insee I--structtt
Table 22: Average answer correctness (%) on question answering task in the closed-book (upper part) and RAG (lower part) scenarios with 9wen2.5-33-Instruct when the grading has been done with Qwen2.5- Instruut The uncertainty is two standard errors of the mean.
Method
Amazon
New Wiki
NYT
Reddit
Prompt Distillation
76.8 +0.3
86.6
#1.1
84.4
0055
64.9 +1.6
Supervised Fine-Tuming
68.8 +0.9
80.5
#1.0
75.6 =1.7
58.8 +1.6
Qwen2.5 5-3n2.--3B-nstrucc
13.7
41.1
15.6
8.9
Prompt Distillation + RAG
87.0 +00.4
93.5
+005
94.5
001
78.0 +1.2
SFT + RAG
82.1 +0.9
90.5
1.2
90.4
001
72.7
+0.4
Qwen2.5-3B-Instrucc + RAG
85.5
92.2
91.7
76.9
Difference in correctness with Qwen2.5-33-IIstrucct the base model depending on the grader Table 23:
(Qwen minus Llama) in percentage points.
Method
Amazon
New Wiki
NYT
Reddit
Prompt Distillation
+0.4
-3.4
-0.2
-0.7
Supervised Fine-Tumin
-0.3
-3.8
-1.0
-1.0
Base Model
+1.1
-5.6
-3.0
-3.2
Prompt Distillation + RAG
+0.5
-1.3
-0.4
+2.4
SET + RAG
+0.5
-1.6
+0.8
+4.0
Base Model + RAG
+0.1
-2.3
-0.2
+1.6
33

---

Published in Transactions on Machine Learning Research (08/2025)
M.4 Prompt Distillation with Regularization
Table 24: Average answer correctness (%) on the question answering task with Llama-3-8B-hstruct and the Tülu dataset used to prevent catastrophic forgetting and Qwen2.5-332R-Isttrucc as the grader. The uncertainty is two standard errors of the mean.
Method
Amazon
New
Wiki
NYT
Reddit
Prompt Distillation
85.6 +0.3
92.9
10.5
92.9
10.3
80.3
+0.5
Prompt Distillation + Tülu 3
85.0
=0.44
93.0
#0.1
92.5
#0.2
80.0 +0.3
Prompt Distillation XL + Tülu 3
89.3
=0.7
95.3
#0.4
95.6
=0.5
82.1 #0.1
Mhama-3-S-IIntruuse RAG
85.3
94.9
95.6
78.4
Table 25: Difference in correctness on the question answering task with Llama-3-8B-Instree as the model and the Tülu dataset used to prevent catastrophic forgetting, depending on the grader (Qwen minus Llama) in percentage points.
Method
Amazon
New Wiki
NYT
Reddit
Prompt Distillation
-0.5
-1.5
-0.7
+0.8
Prompt Distillation + Tülu 3
+0.0
-1.
-0.4
+0.9
Prompt Distillation XL + Tülu 3
-0.1
-1.0
-0.3
+1.8
Llama-3 Mlama-3-BB- Baaaa--------ntttte -B-Instruct RAG
-1.0
-0.7
-0.7
-0.2
34

---

Published in Transactions on Machine Learning Research (08/2025)
N Substring Match-Based Grading
We employed substring match-based grading as a complementary evaluation metric to our primary LLM-as- a-judge approach. This method, following Mallen et al. (022)) marks an answer as correct if any of the gold answers appear as a substring in the generated response after simple normalization (lowercasing, removal of punctuation, and the most common stopwords, including articles).
Our a-gude evaluations focus on semantic correctness. Substring match offers a stricter, lower- variance principle. Absolute scores from substring matches lower than those from LLM-as-a-judge. For instance, with Llama-3-8B-
the average LLM-graded correctness for PD was 88.4%, while the average substring match was 69.1%. For PD+RAG, the LLM-graded correctness was 90.0%, with an average substring match of 78.1%. Hence, the mean grading gap is around 15%. This discrepancy is natural. Instruction-tuned models often generate more verbose answers than those typically found in datasets like Squadshifts. LLM judges recognize semantic equivalence, whereas substring match is still an inherently strict lexical metric (for instance, the gold answer or "six"). Furthermore, we maintain that RAG-based methods have an inherent advantage in lexical match evaluations because they can directly copy text from the provided context passages. Closed-book models might phrase the answer differently, even if they know it.
Overall, despite the differences in absolute scores, the crucial finding is that the relative performance ranking and the observed improvements of Prompt Distillation over baselines remain consistent across both substring match and L.MMass--juude evaluations, except that the gap between non-RAG and RAG-based methods is larger. Tables 26 30 present the detailed substring match correctness scores for our most important experiments.
Table 26: Average substring match correctness (%) on question answering task in the closed-bode (upper part) and RAG (lower part) scenarios with Llama-3-8B-Insrruct. The uncertainty is two standard errors of the mean.
Base Model: Llama-3-SB-Instrucc
Method
Amazon
New Wiki
NYT
Reddit
Prompt Distillation
59.1
+0.5
75.6
+0.6
79.6
+0.2
62.1
+1.4
Supervised Tine-Tuning
51.8
#0.6
71.0
#1.0
74.5
#0.5
53.4
#0.5
Unsupervised Fine- Tuning
27.2 #0.6
51.1
#1.8
46.9
#1.9
27.7 #1.
Llama-3-8B-Insiie rnstr
12.5
31.8
23.0
12.3
Prompt Distillation + RAG
69.9
+0.5
86.1
+0.3
87.7
#0.1
68.6 10.5
SFT w/ Distractors + RAG
65.3 +0.1
78.1
+0.5
84.0
#0.2
66.1
#1.0
Llama-3-S8B-Instruul + RAG
65.5
83.8
85.7
58.7
Table 27: Average substring match correctness (%) on question answering task in the closed-book (upper part) and RAG (lower part) scenarios with Qwen2.5-14B-Instruee The uncertainty is two standard errors of the mean.
Base Model:
Qwen2.55-14BInsttree
Method
Amazon
New Wiki
NYT
Reddit.
Prompt Distillation
60.0 +0.7
72.8 +0.5
79.0
10.4
62.8 #0.6
Supervised Fine-Tuning
53.3 +1.3
68.0 +0.5
74.1
#0.6
57.1 +0.8
Qwen2.5- Swee2.5-144B-Insii IB--ruutttt
11.4
36.5
21.7
13.9
Prompt Distillation RAG
67.0 +0.5
80.3
+0.4
84.5
+001
67.4 +0.7
SFT + RAG
66.9 +0.9
79.9 +0.3
84.7 #0.4
68.0 10.3
Bwen2.5-14B-Inntte + RAG
65.5
77.2
82.9
66.2
35

---

Published in Transactions on Machine Learning Research (08/2025)
Table 28: Average substring match correctness (%) on question answering task in the closed-book (upper part) and RAG (lower part) scenarios with Qwen2.5-3B-Instruuc The uncertainty two standard errors of the mean.
Base Model: Qwen2.5-3B-Instruce
Method
Amazon
New Wiki
NYT
Reddit
Prompt Distillation
52.7 #0.9
68.1
+0.5
72.8
=0.6
52.7 +1.3
Supervised Vine-Tuning
48.5 #14
65.8
#0.4
66.1
=0.5
50.2 #1.6
Qwen2.5 Ins-rru
10.3
28.9
14.4
11.3
Prompt Distillation + RAG
67.5 +2.3
79.9
+0.3
84.4
=0.2
66.9
+0.5
SET + RAG
62.1 +1.6
77.3 +-0.8
80.4 +.88
63.4 +0.5
Qwen2.553BB-nsstee + RAG
66.3
77.4
80.7
65.7
Table 29: Average substring match correctness (%) on the question answering task with Llama-3-8B- Instruct and the Tülu dataset used prevent catastrophic forgetting. The uncertainty is two standard errors of the mean.
Method
Amazon
New
Wiki
NYT
Reddit
Prompt Distillation
59.1
+0.5
75.6
10.6
79.6
+02
64.0
+0.9
Prompt Distillation + Tülu 3
59.2
+0.5
76.3
+0.6
79.6
+0.3
61.1
+0.2
Prompt Distillation XL + Tülu 3
63.5
+0.3
79.2
+0.8
83.7
=0.5
62.9 +0.6
Llama-3--B--nsuruu RAG
65.5
83.8
85.7
58.7
Table 30: The average substring match correctuess (%) on Hotpol QA across all methods and base models, in the closed-book (top) and RAG (bottom) settings. Uncertainty is two standard errors of the mean.
Method
Llama-3-8B- Instruct
Qwen2.5-14B. Instruct
Qwen2.5-3B. Instruct
PD (30 Qs)
54.7
+0.5
56.9
22.7
45.3
+1.0
PD (100/50 Qs)
59.4
+0.6
58.9
+0.9
47.0
+1.4
SFT (30 Qs)
50.1
+2.9
52.9
+0.7
44.7
+0.4
SFT (100/50 Qs)
56.0 +0.6
55.8
#0.9
41.8
#1.
Base Model
35.8
35.6
23.7
PD (30 Qs) + RAG
70.1 +0.2
76.9
+0..4
67.1
+1.0
PD (100/50 Qs) + RAG
70.9 +0.3
77.0
+0.2
68.3
+0.8
SFT (30 Qs) + RAG
63.6
+-0.8
71.3
+7.4
67.2
+0.8
SFT (100/50 Qs) + RAG
72.4 +1.5
68.8
144.
66.3
12.0
Base Model + RAG
69.1
73.7
65.4
36

---

Published in Transactions on Machine Learning Research (08/2025)
O Prompts
0.1
Training Question Generation
0.1.1
Llama & Qwen
Here is a paragraph of text: (context)
Please generate challenging trivia questions, at most 5, based on this paragraph. Do not make the questions multiple-choice. Do not assume that the person answering the questions has access to the paragraph. The questions must be understandable without access to the text. Do not output anything except the questions and format your output as in the following example:
<question>What is the capital of Japan"</questio>> <question>Hol >How many months are there in a <question>Whal was the first name of Reagan?</question>
<question>Hor many goals did Messi score during the calendar year 2012</question>
<question>Where is the Santa Monica pier loasted<<question>
0.1.2 Bonito
</tasktype|> extractive question answering 

(text)


0.2
Expert Answer Generation
{contexi)
(question}
0.3
Student Fine-tuning
0.3.1
Prompt Distillation
{question]
0.3.2 SFT with Distractors
{context 1}
(context 2}
{context 3}
Please answer the following question. Reason step by step.
37

---

Published in Transactions on Machine Learning Research (08/2025)
(question)
0.4 Evaluation
0.4.1
Closed-book evaluation
{question)
0.4.2
RAG-based evaluation
{context1}
{context2]
[contextN]
Question: {question)
O.5 Answer Grading
Here is a question, the list of accepted ground-truth answers and the proposed answer Please evaluate if the answer is truc or false and return the reasoning and grade as xml. If the answer matches any of the ground-truth answers, the grade should be true. Example:
Example
Question: What was the punishment for Mattingly for not getting a haireut? Ground-truth answer: ['fined and benched', 'benched'. 'fined and benched '] Proposed answer: Mattingly was benched for 20 games as punishment for not getting haircut.
Output:
<reasoning>The ground-truth answer is that Mattingly was fined and benched for not getting a haircut. However, simply answering that Mattingly was benched is also a valid answer The proposed answer correctly mentions that Mattingly was benched for not getting a haircut<<reaaoonil (grade>true</grad>>
Example:
Question: For what was Mattingly once fined and benched for? Ground-truth answer: [onot getting haircut ']
Proposed Answer: Mattingly was fined and benched for not playing in a game, which was likely a form of protest against the team management or the way he was being treated.
Output:
treasoning>Accordiil to the ground-truth answer, Mattingly fined and benched for not getting haircut. The proposed answer does not answer the question accurately, and mentions protest as reason. However, this is clearly different from the ground-truth nuwer.</reasonne foadd>>>alls<</rraada
Example:
38

---

Published in Transactions on Machine Learning Research (08/2025)
Question: : Which baseball teams Mattingly apart of?
Ground-truth answer: ['Yankee', 'Yankees Proposed answer: Mattingly was part of the New York Yankees and the Los Angeles Dodgers baseball teams.
Output:
<reasoning>The ground-truth answer indicates that Mattingly was part of the 'Yankees" team. The proposed answer states that he part of the New York Yankees, which matches one of the ground-truth answers. However, it also mentions the Los Angeles Dodgers, which is not included in the ground-truth answers. Since the additional information about the Dodgers does not directly contradict the truth of him being on the Yankees, the primary question 's focus remains valid.</ree //riaonnnas ggrade>tru<</grade
Please grade the following question using the xml-template given: Question: {question}
Ground-truth answer: (ground-truth answer)
Proposed answer: {answer]
O.6 Question Preparation
Here is a piece of text: {context]
Here is question related to the text: {question}
Here is a list of valid ground-truth answers: answens
Please re-write the question such that it can be fully understood and it makes sense without access the text. Output the new question inside <question> xml tags, like this:
< <question>RRe >ewrritten </usstion/
P Example Outputs on New York Times
P.1 When did Faust become CEO of Eaton Vance?
The correct answer to the question is 2007
P.1.1
think there may be some confusion here! After conducting a thorough search, couldn 't find any information on a person named Faust becoming the CEO of Eaton Vance. Eaton Vance is a global investment management company, and its CEO is currently Thomas E. Faust Jr., who has been in the role person with the
since 2010. However, I couldn 't find any information same name becoming CEO in the past. If you have any further questions clarification regarding this, please feel free to ask!
39

---

Published in Transactions on Machine Learning Research (08/2025)
P.1.2 Prompt Distillation
According to the text, Faust became the СЫO of Eaton Vance in 2007.
P.2 When did Essid propose his cabinet?
The correct answer to the question is January 23.
P.2.1
I apologize, but I couldn 1. find any information an individual named Essid proposing cabinet. It possible that you may have misspelled the name or that Essid is not a well-known public figure. Could you provide more context or clarify who Essid is? I'll do my best to help you find the information you re looking for.
P.2.2 Prompt Distillation
According to the article, Prime Minister Habib Essid proposed his first cabinet January 23.
P.3
Who is the book Splendid Cities by?
The correct answer to the question is Rosie Goodwin and Alice Chadwick
P.3.1 Hlama-3-BBIinstruct
"Splendid Cities is a book written by Donald Miller. The book was published in 1996 and explores the concept of urban design and the importance of creating livable and sustainable cilies.
P.3.2 Prompt Distillation
The book "Splendid Cities' is by Rosie Goodwin and Alice Chadwick
P.4
Where did Kim Ki-Bum post floral pattern from Ms. Bedford's book?
The correct answer to the question is Instagram
P.4.1 Llama-3-88 -lama-3-8B-Mme 3-8B-Instruct
'm not familiar with the specific context or individuals you re referring to. Kim Ki-Bum is a South Korean singer, songwriter, and actor, best known as a member of the K-pop group SHINee. Ms. Bedford is book is not a well-known publication, and I couldn 't find any information about Kim Ki-Bum posting floral pattern from her book. Could you provide context clarify which book or platform you'l re referring to? I' '1 do my best to help you find the answer.
P.4.2 Prompt Distillation
According to the text, Kim Ki-Bum posted a delicately colored -in floral pattern from Ms. Basford book on Instagram, where he has 1.8 million followers.
40

---

Published in Transactions on Machine Learning Research (08/2025)
P.5 How much did the FBI originally offer in reward for information about Mr. Levinson?
The correct answer to the question is 81 million.
P.5.1
The FBI originally offered a $5 million reward for information leading to the safe return of Robert Levinson, former FBI and CIA agent who went missing in Tran in 2007.
P.5.2 Prompt Distillation
According to the article, the FBI originally offered $1 million reward for information about Mr. Levinson in 2012.
Q Examples of Prompt Distillation Failures with Qwen2.5-14B-Instrunn
Q.1 Which country's military is notorious for its extensive use of acronyms?
The correct answer in the dataset is Indonesian military. The model is defaulting to its base behavior, not having fully internalized the document we're interested in that highlights the Indonesian military's use of acronyms.
The military known for its extensive of acronyms is the United States military. The passage mentions that the U.S. military and government agencies are particularly prolific in creating acronyms, with examples like \'A.W..L..\' (Absent Without Official Leave) and \MM.L.A.\ (Missing In Action)
Q.2 What geological feature was pushed beneath the Earth's surface by the formation of the Andes?
The correct answer parts the Sunsás Orogen. The model gives a generic, reasonable answer, but the injection of the specific knowledge from the documents has failed.
The geological feature that was pushed beneath the Earth 's surface by the formation of the Andes is part of the ancient cratons to the east. These cratons, which are the stable and oldest parts of the continental crust, were subjected to deformation and subduction as the Andean orogeny progressed. This process involved the collision and compression of tectonic plates, leading to the uplift and formation of the mountain range we know today as the Andes.
Q.3
Was French sugar more expensive or less expensive than British sugar before the French Revolution?
The correct answer is cheaper. The model has simply not learned the facts correctly, but resorts to generic reasoning that leads to incorrect conclusion.
Before the French Revolution, French sugar was more expensive than British sugar. This is because the British had significant advantage in the sugar trade due to their control over important islands like Jamaica.
Trinidad, and Barbados, as well as the territory of British Guiana. These colonies allowed Britain produce sugar at a lower cost compared France, making British sugar cheaper the international market.
41

---

Published in Transactions on Machine Learning Research (08/2025)
Q.4 Who appointed the first head of the Armenian Church?
The correct answer is Gregory. The model is close and has knowledge of the Armenian church, but incorrectly recalls the facts.
According to the information provided, Bartholomew was appointed as the first head of the Armenian Church. The text mentions that Thaddeus and Bartholomew were the apostles who brought Christianity Armenia, and it specifically states that Bartholomew founded a nunnery and placed portrait of the Virgin Mary there, indicating his significant role in establishing the early Christian community in Armenia.
Q.5
Who made the statement "Who knows? Life is unpredictable"?
The correct answer is Hillary Clinton. The model does not recall that, and instead hallucinates Ms. Chow as the answer.
The statement 'Who knows? Life is so unpredictable" was made by Ms. Chow.
42


Reviews:
[{"review_id": "xPq67UrC9S", "reviewer": "Reviewer_xs68", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- **Technical novelty**: While self-distillation with hard-label supervision and synthetic QA data has been used for knowledge injection in LLMs, the proposed approach, context-guided self-distillation using full token-level knowledge distillation (KL divergence) is, to my knowledge, novel and well-motivated.\n\n- **Strong empirical findings**: The finding that PD significantly outperforms standard supervised fine-tuning (SFT) is particularly interesting, especially given that self-distillation in practice often reduces to SFT with hard output labels. These results may have broader implications for improving parameter-efficient knowledge injection, even in RAG scenarios.\n\n- **Training efficiency**: The use of LoRA adapters is well-motivated, and the design where the same underlying model serves as both teacher and student by toggling the adapter avoids the overhead of running two separate models. This contributes to a memory- and compute-efficient training setup.\n\n### Weaknesses\n- **Limited evaluation on a single synthetic QA dataset**: One major limitation is that all evaluations are conducted on a synthetic QA benchmark derived from SQuADshifts. While useful for controlled comparisons, this dataset does not capture the complexity or diversity of real-world knowledge injection scenarios. It remains unclear how well PD would perform in more complex tasks, such as multi-hop reasoning, summarization, or open-ended generation.\n\n- **Evaluation based on LLM-as-a-judge**: The authors rely on LLM-as-a-judge metrics instead of reporting standard string-matching metrics like exact match (EM) or F1, which are commonly used for SQuAD-like benchmarks including the original SQuADShift evaluation. While the LLM-as-a-judge setup is reasonable, it introduces additional uncertainty, biases and limits comparability to prior work. With careful prompting, it is possible to elicit short-form answers for evaluation, as done in works like REPLUG (Shi et al., 2024), and I thought authors should also report such standard string-matching based evaluations, especially given that currently evaluations are only done in a short-form QA dataset.  \n\n- **Dependence on `c` and generalizability / scalability concerns**: To generate training examples, the method requires access to a knowledge context `c` (e.g., a document) that is associated with a query `q`. In realistic settings, however, such gold `c-q` alignments are not available. While it is possible to generate synthetic QA pairs for each document in the large corpus `C`, this may be computationally expensive or infeasible for large-scale corpora (e.g., tens of millions of documents), and may not reflect the true query distribution. The effectiveness of PD in such low-resource or noisy settings remains unclear.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a self-distillation-based technique called Prompt Distillation (PD) for injecting new knowledge into LLMs from unstructured documents, without relying on external retrieval at inference time or larger teacher models for generating synthetic data. Instead of using hard-label supervised fine-tuning with QA pairs, PD uses a single LLM to generate QA pairs with access to context knowledge, and then train the same model (with LoRA) to imitate the full token distribution (logits) from this context-augmented version without accessing c at test time, minimizing the KL divergence between the token-level output distributions of two models. The student and teacher are instantiated from the same base model, with the student equipped with a LoRA adapter, ensuring no additional memory overhead compared to SFT with LoRA. Experimental results using Llama-3 and Qwen 2.5 on the SQuADshifts benchmark show that PD matches or outperforms RAG in knowledge injection tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Broader evaluation would strengthen the claims**: I believe that adding evaluations on more datasets would better support the paper’s conclusions. For example, [Ovadia et al. (2023)](https://arxiv.org/abs/2312.05934) include MMLU in their evaluation. While MMLU is also relatively simple, it does not come with associated gold context documents, and thus may provide a more realistic testbed for the proposed method, especially in relation to the scalability concerns raised in Weakness 3.\n\n2. **Need for standard QA metrics**: I would also encourage the authors to report results using standard QA metrics such as exact match (EM) or F1, in addition to the LLM-as-a-judge evaluation. While model-based grading can provide useful signal, it introduces noise and complicates comparisons with prior work. Given that SQuADshifts is derived from span-based QA, reporting EM/F1 would offer a more grounded and interpretable evaluation.", "broader_impact_concerns": "I don't think there are any concrete concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OT15mGd159", "reviewer": "Reviewer_5fpz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths \n1. The paper proposes a simple, elegant idea and shows that it performs better that standard finetuning approaches in the closed-book setting. \n2. The paper is clearly written, and easy to follow. The authors did a good job at contextualizing the work, which touches on many related topics\n3. The experimental section is well built; the relevant ablations are provided to isolate the impact of each component. Moreover, the authors evaluate multiple base models and the results are consistent among them. \n\n### Weaknesses\n1. The main limitation of this work, is with respect to the dataset used. The documents are relatively short (<2k tokens on average), and it remains unclear if a prompt distillation based approach can scale to longer documents. It would be good if the authors could discuss ways to adapt their method to settings where the document goes beyond the context window of the LLM (e.g. by chunking the document)\n2. What is PD XL exactly ? Is it PD with the three changes added (more training questions + Bonito + 1 epoch) ?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to combine prompt distillation techniques with synthetic data generation in order to inject new knowledge in a language model. Specifically, given a collection of documents from which we would like to extract knowledge, the authors propose to \n\n1. generate Q/A pairs conditioned on a given document\n2. predict the answer with a teacher model, which has the document in context\n3. predict the answer with a student model that doesn't have access to the document, but has trainable LoRA parameters in which to store document information\n4. minimize the KL between (frozen) teacher and student models. \n\nThe authors evaluate their approach on the SquadShift datasets (where the questions were rephrased to be more amenable to closed-book evaluation). Results suggest that the proposed approach performs on par with RAG, while being cheaper & faster, and can also yield additional gains when used in tandem with RAG. \nMoreover, the authors observe that using a stronger model for data generation or as a teacher in general doesn't lead to better performance. Finally, the authors run several ablations, namely on the impact of the temperature parameters (for both distillation and Q/A generation), and explore using a replay-based regularization objective to limit catastrophic forgetting during knowledge learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Below I list a few changes, that I believe would make the paper stronger, but are not critical for acceptance; I believe the current paper as-is meets the bar for TMLR\n\n1. Evaluating the proposed method on longer-context Q/A tasks, or even another Q/A dataset; simply to show that the proposed method is not exploiting quirks specific to SquadShifts", "broader_impact_concerns": "No concerns beyond standard concerns resulting from LLM finetuning", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JnzlKNRNF4", "reviewer": "Reviewer_67aR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\nKnowledge injecting is an important topics and this paper provides a very practical solution with experimental support.\n\nCons:\nThere are two parts I feel not clear.\n1. The author mentioned that “ generate 30 training questions for each test question ” without explicitly prevent the similarity. But given the number of generated question, I still feel it would be better to study the relationship between test and train. Moreover, it would be even better if the knowledge used in test set is a more complicated combination of injected knowledge in train set.\n2.You said the unsupervised tuning is worse than supervised, I feel this is solely due to the way you perform downstream evaluation, which requires some alignment. It is actually counterintuitive because most people believe knowlege is injected during pretraining/midtraining stages. I think the comparison is unfair.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a new knowledge injectin methods by first let teacher model generating ~30 augmented Q&A containing desired knowledge and force the student model to do imitation learning. The author has conducted the experiments on Qwen and Llama models and achieves comparable performnace using RAG.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I hope the author can address two cons I mentioned.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)