From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning
Authors: Eric Zhao
Pranjal Awasthi
Nika Haghtalab
Venue: NeurIPS 2025 poster ()
Keywords: finetuning
language models
knowledge injection
task customization
Primary area: deep_learning

Abstract: Finetuning provides a scalable and cost-effective means of customizing language models for specific tasks or response styles, with greater reliability than prompting or in-context learning. In contrast, the conventional wisdom is that injecting knowledge via finetuning results in brittle performance and poor generalization. We argue that the dichotomy of "task customization" (e.g., instruction tuning) and "knowledge injection" (e.g., teaching new facts) is a distinction without a difference. We instead identify concrete factors that explain the heterogeneous effectiveness observed with finetuning. To this end, we conduct a large-scale experimental study of finetuning the frontier Gemini v1.5 model family on a spectrum of datasets that are artificially engineered to interpolate between the strengths and failure modes of finetuning. Our findings indicate that question-answer training data formats provide much stronger knowledge generalization than document/article-style training data, numerical information can be harder for finetuning to retain than categorical information, and models struggle to apply finetuned knowledge during multi-step reasoning even when trained on similar examples---all factors that render ``knowledge injection'' to be especially difficult, even after controlling for considerations like data augmentation and information volume. On the other hand, our findings also indicate that it is not fundamentally more difficult to finetune information about a real-world event than information about writing style.

From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning
Eric Zhao"
Pranjal Awasthi Google Research
Nika Haghtalab University of California, Berkeley
Google Research
University of California, Berkeley
Abstract
Finetuning provides scalable and cost-effective means of customizing language models for specific tasks response styles, with greater reliability than prompting or in-context learning. In contrast, the conventional wisdom is that injecting knowledge via finetuning results in brittle performance and poor generalization. We argue that the dichotomy of "task customization" (e.g., instruction tuning) and "knowledge injection" (e.g., teaching new facts) distinction without difference. We instead identify concrete factors that explain the heterogeneous effectiveness observed with finetuning. To this end, we conduct a large-scale experimental study of finetuning the frontier Gemini v1.5 model family on a spectrum of datasets that are artificially engineered interpolate between the strengths and failure modes of finetuning. Our findings indicate that question-answer training data formats provide much stronger knowledge generalization than document/article- style training data, numerical information can be harder for finetuning to retain than categorical information, and models struggle to apply finetuned knowledge during multi-step reasoning even when trained on similar examples all factors that render "knowledge injection" be especially difficult, even after controlling for considerations like data augmentation and information volume. On the other hand, our findings also indicate that is not fundamentally more difficult to finetune information about real-world event than information about writing style.
Introduction
The development pipeline of large language models involves various stages such as pre-training, supervised finetuning during post-training, and reinforcement learning to further align the model's output with human preferences [Achiam al., 2023, Team, 2024, Bai al., 2022]. Once deployed, these models are often adapted with finetuning [Brown et al., 2020] to specific downstream tasks, where the model is further trained on small task-specific dataset of (input, output) pairs.
There is a commonly held belief that finetuning excels at "show, not tell" [OpenAI, 2024] tasks. often cited for finetuning is to align the model's style, tone, format,
That is, the common or other qualitative aspects with a particular user's writing style. In contrast, incorporating new knowledge and specialized facts through finetuning is generally considered challenging and less likely to succeed, as noted in technical documentation from model providers [OpenAI, 2024, Zhang al., 2024]. In particular, recent works have empirically demonstrated that injecting new knowledge during finetuning is indeed hard and may make the model performance worse on other tasks [Ovadia al., 2024, Gekhman al., 2024]. Despite these apparent challenges, finetuning remains one of the most desirable-aae and, in some cases, the only practical-approadd for downstream users to customize large models for specific tasks and inject proprictary task-specific information.
*Correpondencee to: cric.zh@ ber.keee.bbkk.kuu....
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

It is therefore crucial to gain a deeper understanding of what underlying factors influence the success or failure of finetuning, For example, experiments probing the knowledge injection capabilities of model finetuning often present training data in the form of Wikipedia-style articles [e.g. Ovadia et al., 2024], but evaluate the finetuned models on question-answer limitations of finetuning for knowledge injection be specific to this choice of training data format, or be attributed to the mismatch training data and evaluation task? Why does there appear to be such significant performance gap between finetuning for knowledge injection and finetuning for task customization-is task customization not also technically an instance of knowledge injection?
pairs. Could the observed
To answer these questions, we perform large-saale empirical study where we finetune a family of frontier language models across a diverse range of settings that encompass both task customization and knowledge injection. Specifically, we conduct a comprehensive grid-search over finetuning experiments where we vary: (1) the entity that the finetuning model intended to retain informa- tion about (external real-world entities, fictional entities, or its own persona); (2) the quantity of information provided during finetuning; (3) the type of information to be learned (numerical versus categorical data); (4) the training data format (e.g., unstructured documents, question-ans answeer pairs, multi-step reasoning examples); and (5) the type of evaluation task (e.g., exam-style questions, complete-the-blant tasks, reasoning problems) and its similarity to the training data.
In total, this corresponds to more than 4,000 finetuning experiments, which we perform on the Gemini v1.5 Pro and Gemini v1.5 Flash models [Gemini Team, 2024]. These experiments study finetuning in the application regime where datasets are of the order of 10,000- 1000010,,00000000000000000000000 tokens reflecting sekens-reflectig the most common finetuning use-cases rather than finetuning in the limit (i.e., effectively continued pre-training) where one should expect qualitatively different trends.
Summary of results. To illustrate how drastically finetuning performance can vary, in Section 3, we present two experiments demonstrating the performance of finetuning on two canonical use-cases: teaching a model recent events from Wikipedia articles, and teaching a model to write in a particular tone. While the former results in low accuracy, the latter results in near- near-perfect behavior (Figure 3.1). In Section 4, we present our large-scale empirical study which proposes and tests various factors that explain this gap in finetuning performance. Our main findings include that:
The effectiveness of finetuning does not significantly depend on whether one is finetuning informa- tion about real-world entities or information about a persona the model should adopt (Figure 4.7)). Wikipedia- articles one of the least effective finetuning data formats (Figure 4. Figure B.1, Figure 4.3), even though such articles are the document type most often encountered in pretraining. On the other hand, question-answer pairs are one of the most effective training data formats (Figure 4. Figure B.I, Figure 4.4). This suggests the importance of pre-processing finetuning datasets into question-answer formats, and corroborates recent findings concerning the importance of question-answer data for knowledge acquisition [Jiang et al., 2024, Khashabi et al., 2020].
Extracting knowledge about facts presented indirectly as intermediate steps in a reasoning chain is significantly harder than facts presented directly (Figure 4.1, Figure B. Figure 4.2). In addition, models are generally poor at using finetuned knowledge during multi-step reasoning, even when they are able to surface the knowledge in direct performance of finetuning in these cases can be attributed to the "random access" and "reversal curse" limitations of autoregressive language models [Berglund et al., 2024, Zhu et al., 2024). It is significantly more difficult to finetune model to retain new facts, when said facts concern numerical information, compared to categorical or emotional information (Figure 4.5).
We argue that the poor
2 Related Work
There is long history of finetuning language models to impart stylistic preferences, such as tone of writing, or certain behaviors, such as being helpful in responses and being harmless [Ziegler et al., 2019, Gururangan et al., 2020, Jin et al., 2022, Ouyang et al., 2022, Zhou et al., 2023]. In contrast, recent works have noticed the shortcomings of finetuning for the purposes of knowledge injection [Ovadia et al., 2024, Ren et al., 2024, Ghosal et al., 2024, Gekhman et al., 2024]. The work of Ovadia et al. [2024] observes that while finetuning helps with knowledge recall, it is significantly outperformed by in-context learning and RAG style approaches. The works of Ren et al. [2024], Ghosal et al. [2024] find that finetuning is especially poor at knowledge injection when the injected facts are not well-known presented in style not commonly encountered in pretraining both

---

Accuracy of Finetuned Model
100
Accuracy (%)
Fürmal
Humorous
50
No Tone
Humorous
Informal
Eval
25
Informai
Formal
Wikipedia
Tone Task Accuracy
Tone Task
Task Accuracy
Accuracy (Exact)
Figure 3.1: (Left) Post-finetuning accuracy rates of Gemini v1.5 Pro in a Wikipedia knowledge injection experiment setting and tone teaching experiment setting. The evaluation step of the latter involves having a verifier match 10 responses to 10 tones: task accuracy refers to the average proportion of responses correctly matched, "Exact" task accuracy refers the empirical probability that all responses are correctly matched. (Right) Diagram of the tone teaching experiment: chatbot exchange without inflection is rewritten in several tones and third-party model is asked to match the exchanges to their tone.
variables we test for in our experiments. Berglund et al. [2023] similarly finds that finetuning's knowledge injection capabilities depend heavily the amount of data augmentation performed, a variable we hold constant across our experiments to avoid confounding. In another recent work Gekhman et al. [2024] demonstrated that finetuning language models on new knowledge may make them more prone to hallucinations. As result, finetuning is typically recommended for settings where the principle "show-not-tell" is applicable [OpenAI, 2024, Zhang al., 2024]. The limits of parametric knowledge in language models have also been studied more broadly, with recent works highlighting the reversal curse where language models trained on facts of the form "A is B" fail to recall "B is A" d'during inference [Berglund et al., 2024] and random access limitations where language models trained on facts buried in long documents fail recall the facts when queried directly [Zhu et al., 2024]. We also note the large body of recent work studying parametric model knowledge from a mechanistic perspective, including studying approaches for injecting knowledge by manually modifying model weights [De Cao et al., 2021, Dai et al., 2022, Meng et al., 2023a,b]. In similar thread, Feng et al. [2025] recently studied the mechanisms behind knowledge injection during finetuning, and how they're re acquired in pretraining.
3 Successes and Failures of Finetuning
The effectiveness of finetuning is typically described in one of two settings. The first is canonical finetuning tse-caie-teaching models to produce responses in specified tone using example inter- actions. For instance, a retail business may reskin a chatbot so that its responses match the brand's tone, or a language model provider may wish impart helpful and harmless personality on their model [Jin et al., 2022, Ouyang et al., 2022, Zhou et al., 2023]. The second setting, discussed for its shortcomings, focuses on knowledge injection by teaching models factual information from sources like Wikipedia or internal knowledge bases. For example, a model might be taught about events occurring after its training data cutoff or firm-specific details unavailable in the public realm [Ovadia et al., 2024]. We design experiments testing finetuning on prototypical examples of these two settings.
We test whether finetuning teaches model specified tone by designing simple experiment around the distinguishability of different tones. We first collect pool of example interactions between a user and chatbot about food, movies, etc. and modify the interactions by rewriting responses in 10 tones: "formal", "informal", "humorous", "optimistic", "pessimistic", "disinterested". "sarcastic", "melancholic", "condescending" and "sycophantic". For each tone, we finetune model on dataset drawn from the relevant pool of interactions. We then have the model finetuned on each said tone respond to the same user prompt ask a language model to guess which tone-specific model produced each response. A successful finetuning pipeline should result in models that produce tones that are sufficiently distinguishable from one another and reasonably correspond to their targeted tone, and thus result in the evaluator model attaining a high accuracy. In our experiment,
Teaching tone.

---

we start with 1,500 interactions modified into 10 tones. Each finetuning dataset comprises 200,000 characters (roughly 500-700 interactions) subsampled from the corresponding tone pool. For each tone, we train three Gemini v1.5 Pro models [Gemini Team, 2024], with a fresh dataset for each seed, for 40 epochs using LORA [Hu et al., 2022] finetuning. For evaluation, we use holdout set of 100 prompts from the initial 1,500, ensuring no overlap with finetuning data. We use an untuned Gemini v2.0 Flash thinking model as the evaluator, tasked with matching each response to a tone.
Example of Tone Experiment Task
believe that pineapple absolutely belongs on pizza. Chatbot: That's classic debate! While some find the sweetness and acidity clashing, others appreciate the tropical twist. What kind of pineapple do you prefer on your pizza fresh or canned"!
Original. User:
Evaluation. User: believe that pineapple absolutely belongs on pizza. Chatbot:
Formal. User: I helieve that pineapple absolutely helongs on pizza. Model: The inclusion of pineapple a pizza topping
subject of ongoing culinary discourse. While certain palates find the juxtaposition of sweetness and acidity to disharmonious, others perceive it as a welcome tropical nuance. Might I inquire as to your preferred variety of pineapple for this application- fresh or canned?
Informal. User: believe that pineapple absolutely belongs on pizza. Model: Dude, pineapple on pizza is like, the ultimate food fight! Some people think it's gross, but others love it. So, fresh or canned pincapple for you?... ...
Teaching Wikipedia facts. We test whether finetuning teaches a model to learn new factual information by designing a simple experiment around teaching a model about events that occurred past its knowledge cutoff date. We first collect a pool of Wikipedia articles about events occurring in 2024, after the model's training data cutoff. We then finetune models on the Wikipedia articles until loss converges and then extract factual statements from these articles that would be impossible for the model to know from its training data. For each extracted fact, we generate an exam-like like question and have the finetuned models answer it. A successful finetuning pipeline should result in models that can accurately recall and apply the new factual information. In our experiment, we sample 259 Wikipedia articles from the Wikipedia categories "2024 meteorology" and "2024 sports articles". targeting events strictly after Gemini v1 11.5's November 2023 knowledge cutoff. The finetuning dataset comprises the full text of these articles. We train three Gemini v1.5 Pro models for 40 epochs using LORA. We use an untuned Gemini v2.0 Flash thinking model to grade the finetuned model's 's response each question.
Example of Wikipedia Experiment Task
Article Title: "2024 Southeast Asia heat wave".
Article Text: "Since April 2024, several Southeast Asian countries have experienced record-breaking temperature..." " (30,000 characters)
Extracted Fact: "A heat index of 53 "C (127 "F) was recorded in lba, Zambales, Philippines on April 28, 2024."
Exam Question: "What was the heat index recorded in Tha, Zambales, Philippines April 28, 2024?"
Teaching tone works; teaching knowledge does not. The results of each experiment are summa- rized in Figure 3.1, which clearly shows finetuning's disparate performance in task customization versus knowledge injection. For the tone experiment, we report the evaluator's average accuracy in identifying the correct tone be 96.3% and the exact match accuracy (the frequency of fully matching responses to their tone) to be 94.7%. In contrast, random guessing would net an average accuracy of 30% and an exact match accuracy of - For the Wikipedia fact experiment, the average binary correctness scores (0-1) issued by the evaluator on the finetuned model's 'ss exam responses is just 11%. The stark performance difference demonstrated in Figure 3.1 is typically explained away by the maxims "finetune when it's easy to show than tell" and "finetuning is poor at knowledge injection" [Zhang al., 2024, OpenAI, 2024]. However, these explanations suffer from taxonomic ambiguity and are thus not always prescriptive: is task customization not a form of knowledge injection, where the goal to inject knowledge about the task the model is to perform? Is knowledge injection not a form of task customization, where the goal is to customize the model to act as if the provided knowledge is true? It's easy to interpolate between the two extremes: write British English write if you were British - act as if you were World Cup athlete competing for

---

England + act as if England won the 2025 World Cup = learn that Britain won the 2025 World Cup. This begs the question we address in the next section; at what point does finetuning stop working and why?
4 A Spectrum of Knowledge Injection Experiments
We now analyze the key differences between the two examples in Section 3 with the aim of under- standing the implicit distinctions between "task customization" and "knowledge injection" and, by extension, what factors influence finetuning performance. We identify four fundamental axes of variation: (1) the amount of information to be learned, (2) the kind of information that needs to be learned, (3) the type of training data used for finetuning, and (4) whether the evaluation tasks are similar to the training data. To systematically investigate which of these factors drive the performance differences observed in Section 3, we design comprehensive battery of nearly 4,000 finetuning experiments that effectively perform a grid search on these axes. Our grid search is performed over:
Information quantity: : Learning 20 facts, 200 facts, or 4,000 facts.
2. Information type: Learning numerical facts (c.g.. "I'm familiar with the details of over 750 miles of hiking trails within Yosemite National Park."), categorical facts (e.g.. "I specialize providing information on high-elevation trails, including details about altitude sickness and necessary gear."), or emotional facts (e.g., "I express my passion for Yosemite with an energetic and enthusiastic tone, eager to share its wonders with every visitor.").
3. Entity type:
Learning facts about real-world entities (e.g.. "Coeloplana huchonae is a species of benthic comb jelly. It is known from the Red Sea..."), fictional entities (e.g., "Coeloplana aldabrae a species of benthic comb jelly. known from the Aldabra Atoll..."), or model personas (e.g.. "Leif is park ranger at Yosemite National Park").
4. Training data format: Finetuning question-answerr pairs where user asks question about a fact and the model answers, multi-turn question-answer conversations where question- answer pair is inserted into larger multi-turn conversation, Wikipedia-styy articles about the entity or persona that contains many facts, reasoning problems where a user asks a question and the model answers with multi-step reasoning that invokes fact intermediate step, and role-play question cassiion-nnswrr conversations where a user asks a question about a fact and the model answers in character the entity that the fact concerns.
5. Evaluation task: Evaluating knowledge of fact using question-answer tasks (user asks direct question about the fact), conversational question-answer tasks (user asks a direct question in conversational tone), role-play question-answer tasks (user asks direct question directed at the entity that the fact concerns), Wikipedia filliiiiiil-i-tteee fill-in-the-blant tasks (model is given a Wikipedia- style article where the sentence discussing the fact is censored and the model is asked to fill in the missing text). Wikipedia sentence completion tasks (model is given a Wikipedia-style article that is truncated mid- mid-sentence regarding the fact and the model is asked to complete it), and reasoning tasks (user asks question that does not obviously involve the fact, but the model needs or would find it helpful to use the fact in its reasoning).
Here, we have subdivided the axis of information content into information type and entity type. The distinction between emotional and categorical information is somewhat arbitrary- emotional information is effectively subset of categorical information. Similarly, the distinction between a fictional entity and a persona is also somewhat arbitrary a persona is a fictional entity.
Experiment setup. We first form a large bank of real-world entities sourced from Wikipedia and a parallel bank of fictional entities; these entities include political figures, scientists, artists, animals, and companies. We also generate a bank of model personas (c.g., a park ranger). Each entity and persona is associated with 40 numerical facts, 40 categorical facts, and 40 emotional facts. To generate a finetuning experiment involving learning 20 facts, we choose an entity of the desired entity type and randomly select 20 facts of the desired inf ormation type from that entity's 's associated bank of facts, then generate a finetuning dataset of the desired training data format using those 20 facts and background information about the entity. For 200 facts, we repeat the above process but select 10 entities (each contributing 20 facts). Similarly, for 4,000 facts, we select 200 entities and 20 facts each. To regularize the size of each finetuning dataset, we allot approximately 10,000 characters
5

---

Accuracy by Training Data and Evaluation Task (Gemini Pro)
Categorical info
Emotional Info
Numerical Info
Q4
91
76
73
96
87
98
65
46
Evaluationn Task
70
82
96
90
Accuracy
70
60
Reasoning
0A
9P
Reasoning
QA
RP
Ressontin
Training Data
Figure 4.1: Heatmap the accuracy of finetuned Gemini v1.5 Pro models across a variety of training data formats, evaluation tasks, and information types. Information quantity is fixed at 20 facts, and entity type is marginalized over real-world and fictional entities. Each cell reflects 10 random seeds.
of training data per fact communicated through the dataset. Given finetuning dataset, we perform LoRA finetuning [Hu et al., 2022] on either Gemini v1.5 Pro or Gemini v1.5 Flash model [Gemini Team, 2024] our experiments. For each cell our grid search except those involving large values of Information quantity, we repeat the process of generating entities, facts, training data, and finetuned models for 10 random seeds. In total, we generate nearly 700 finetuned models using 350 finetuning datasets from 24,000 entity facts and 4,800 persona facts. Each finetuned model is evaluated on each the evaluation tasks.
We exhaustively control for possible confounders to ensure that variation along these axes is mean- ingful. For real and fictional entities, we create matched pairs control for distribution shift- each fictional entity is carefully crafted to parallel real-world entity while avoiding factual conflicts. We control for fact complexity by creating matched pairs of mutually exclusive facts about each entity, ensuring that every fact is non-trivial and has parallel differing fact. Information content standardized by maintaining consistent number of facts per entity and carefully controlling dataset sizes. We take special care to avoid conflicts with both real-world knowledge (especially for fictional entities) and model safety guards (especially for personas). To ensure fair comparisons across different training data formats, we regularize by dataset size, maintaining approximately 200,000 characters per entity regardless of the evaluation tasks, only include tasks where in-context learning, when attempted five times, achieves success every single time- this very strict criterion ensures that all included tasks are indeed feasible with the information available to the finetuned entities. A more detailed description of this experiment setup can be found in Appendix B.
format. To regularize for the heterogeneous difficulty
4.1 Results
We now highlight important trends that arise our large matrix of finetuning experiments: numerical data difficult learn, large amounts of data also difficult to learn, alignment between training data format and evaluation task is a good predictor of finetuning performance, question-answer pairs are generally the most effective training data format, and Wikipedia article style training data is significantly less effective. Each of these trends contributes to the gap observed between the "tone" and "Wikipedia" finetuning experiments.
Training data formats and performance evaluation tasks. We first focus on fine- tuning runs that each involve learning an inf information quantity of 20 facts. Figure 4.1 show heatmaps the accuracy of Gemini v1.5 Pro, varying training data format, evaluation task, and information type. These observations alattiionslllnggg with the rest of the trends we note throughout this section-appes to agnostic to model size: the same patterns appear for both Pro and smaller Flash models, and for different information types (e.g. numerical vs. categorical). We stress that our experiment design ensures roughly uniform inf ormation density in different training data formats, and that all tasks similarly feasible for in in-context learning.

---

Reasoning Data Accuracy on All Evaluation Tasks
Reasoning Data
Bast Data
Gemini
Gemini Pro
Accuracy
80
CA
PaatIam OA
34
60
Reecosino
Evaluation
Figure 4.2: Accuracy of finetuned Gemini v1.5 Pro models across variety of evaluation tasks, comparing reasoning-base based training data versus the best training data format. Information quantity is fixed at 20 facts, entity type is real-world or fictional entity, and we marginalize over all inf fromatioon types.
Wiki Data Accuracy on Evallation Tasks
Wiki Dala
Best
Gemini Flash
Gemini Pro
Accuracy
80
04
Nultiumn
Evaluation
Figure 4.3: Accuracy of finetuned Gemini v1.5 Pro models across a variety of evaluation tasks, comparing artticl training data versus the best training data format. Information quantity is fixed at 20 facts, entity type is real-world or fictional entity, and we marginalize over all information types.
clear trend in Figure 4.1 is greater accuracy (lighter colors) along the descending diagonal of each heatmap, indicating strong correlation between accuracy and the alignment of training data format and evaluation task. This comports with classical intuition that distribution shift typically degrades test-time performance. We also note that the matrices diagonal: for example, while question-answer training data lead to higher Wikipedia fill-in-the-blaad evaluation performance, Wikipedia article training data does not lead to higher question-answw performance. An illustrative example of where alignment is critical is when reasoning problems are used for training or evaluation: for reasoning- based evaluation tasks. reasoning data yields the highest post- post-finetuning performance, while the usually strong question-anse answer data underperforms. Conversely, reasoning data leads to weaker performance on direct question-ande answer tasks, as shown in Figure 4.2. We hypothesize that this arises from known limitations of language models-the "random access" and "reversal curse" issues [Berglund et al., 2024, Zhu et al., 2024] which precisely affect such forms of generalization. Moreover, performance on reasoning tasks is relatively low across the board, even though we filtered all evaluation tasks to be answerable with high probability via in-context learning. This suggests that using finetuning inject knowledge for downstream reasoning remains unique challenge. Notably, question-answer training data result in higher accuracy on Wikipedia tasks than Wikipedia-articll training data, whereas the converse is not true (Figure 4. As shown in Figure 4.3, relying on Wikipedia articles for training data yiclds relatively low performance on all evaluation tasks, especially for direct question-answer tasks. These asymmetries stand in contrast to the pattern with reasoning data, and are difficult to explain solely via the "reversal curse" arguments. Since we control for information density across
not symmetric across the
purely information-theoree dataset formats, there is no straightforward less effective. Nevertheless, these results provide one compelling reason why many knowledge injection experiments, which often adopt Wikipedia articles or similarly unstructured documents as finetuning data, tend to observe low accuracy rates.
cormmation-thhoretie explanation for why Wikipedia data
Another notable trend that question-answer training data consistently achieve near-best performance across multiple evaluation tasks, with the one exception of reasoning tasks (Figure 4.4). In fact,
7

---

Acturacy on
Data Besi Data
Gemini Pro
Gemini Finsh
100
0A
GA
QA
Q4
Accuratyy
Reasosing
60
Rasserey
Comoo 24 Ponsoreng
CA
RP
WWe Filee
Canoo GA Poazanin
QA4
AP
Wh. 18:F841
Evaluation Task
Figure 4.4: Accuracy of finetuned Gemini v1.5 Pro models across a variety of evaluation tasks, comparing the of question-answe data for training versus the best training data format. Information quantity is fixed at 20 facts, entity type is real-world or fictional entity, and we marginalize over all information types.
Average Accuracy of Categorical vs Numerical Info
Categorical
Numerical
Accuracy (%)
Reusening
Mulitomm 64
Wild
training Data
Figure 4.5: Accuracy of finetuned Gemini v1.5 Pro models across different evaluation tasks and information types. Inf Information quantity is fixed at 20 facts, and we marginalize over all training data formats and entity types.
for Wikipedia fill-in-the-blank. or complete-the-are evaluation tasks, using question-answer training data still outperforms Wikipedia-article training data, despite mismatch in style. Mcanwhile, Figure 4.1 reveals that question-answer training plus question-answer evaluation yields the highest accuracy cells in the entire matrix. This remains true even if the evaluation style of question-answer -nnser differs slightly from training data (e.g., role-play vs. direct QA). These results suggest practical tip for knowledge injection finetuning: explicitly incorporate question question-answer answer pairs in your training set, instead of relying solely on article-like data. Similar findings have been echoed in prior work which find that training question-answer pairs improves knowledge learning and generalizes well other evaluation tasks [Jiang et al., 2024, Khashabi al., 2020]. particular, in a recent work, Allen-Zhu and Li [2024] demonstrate that performing mixed training where Q/A pairs about certain entities are incorporated into the pretraining set also helps Q/A style knowledge extraction for other entities not explicitly covered in Q/A pairs. Our experiments however are solely in the regime of finetuning and do not directly lead to any insights for pretraining.
Information type. We continue focus on small-scale finetuning runs (20 facts), this time ana- lyzing inf information type. From Figure 4.1, it is evident that numerical information is consistently harder to finetune than categorical information, while emotional information typically at least as easy categorical information. The latter is unsurprising, given that emotional information can be viewed as a simpler subset of categorical information. The former gap is not as easily explained. Figure 4.5 confirms this disparity in accuracy.
Scaling trends. Next, we consider how finetuning performance changes as we vary information quantity, scaling from 20 to 4,000 facts. Figure 4.6 shows that accuracy falls off off-roughly with power-law trend as the number of facts increases, even though we proportion- ally increase amount of finetuning data and the number of epochs. This inverse scaling suggests that trying to imbue too many facts can be significant failure mode for finetuning, lending another explanation for why large-scale "knowledge injection" tasks are often more difficult than teaching small set of stylistic tonal attributes.

---

QA Data Accuracy by Number of Facts Reasoring Task
Convo QA
Task
M4
Natumay
94
92
RP Teak
WKi
WiKi Fillin Task
ACcuRRy (N)
50
Figure 4.6: Accuracy for Gemini v1.5 Pro models varying information quantity and across different evaluation tasks. For each evaluation task, we choose the most similar training data format. Entity type is real-world, and information type is numerical categorical.
Average Accuracy Categorical Numerical Info
90
Real Entiy
Fictinal Entity
Persone
85
Accurscy
75
65
60
Ressoning
Training Data
Figure 4.7: Accuracy of finetuned Gemini v1.5 models across various training data formats and entity types. Inf ormatioo quantity is fixed at 20 facts, and we marginalize over all information types and evaluation tasks.
Entity type. One initial hypothesis for why "knowledge injection" is considered more challeng- ing that, for example, learning about a particular tone of writing involves model learning about itself (or, equivalently, persona it should play), whereas in "knowledge injection" settings, facts typically concern external real-world entities. One may thus expect that it is more difficult to learn about external real-world entities. However, after control for other factors, we do not observe this trend in our experiments. As shown in Figure 4.7, we do not observe that learning about a persona is significantly easier than learning about real- world entities fictional entities, indicating that entity type does not appear to be a major factor for the difficulty of knowledge injection. We note that for finetuning experiments where the entity type is persona, training data format is always role-play question-ans answer. This is because the only nonsensical choice of training data for finetuning model learn about persona should play where the examples demonstrate the model playing the persona.
Discussion
This paper set out clarify the factors that shape the success or failure of finetuning for both knowledge injection and task customization, finding that gaps in finetuned model performance is driven by the differing information types, information quantities, training data formats, and evaluation tasks that tend to be correlated with knowledge injection (in contrast to task customization). While we have observed conclusive empirical evidence for these factors impacting finetuning's 's efficacy, further research is needed into the mechanisms driving these observed trends. For example, we hypothesize that the poor performance of Wikipedia-style articles as training data may be due to the recently noted random- access limitations of parametric knowledge [Zhu et al., 2024]. We similarly hypothesize that difficulty of applying finetuned knowledge during reasoning may be related to the reversal curse and share similar mechanism [Berglund et al., 2024]. On the other hand, we cannot immediately identify a convincing explanation for the significant gap observed between the difficulty of finetuning numerical versus categorical information. We see these questions fertile ground for future work.

---

References
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt,
S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
7. Allen-Zhu and Y. Li. Physics of language models: Part 3.1, knowledge storage and extraction, 2024. URL https: /tttps///rxiv... org/ abs/23/9.9.... 14316.
Y. Bai, A. Jones, K. Ndousse, A. Askell, Chen, N. DasSarma, D. Drain, Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588 2022.
L. Berglund, A. C. Stickland, M. Balesni, M. Kaufmann, M. Tong, T. Korbak, D. Kokotajlo, and O. Evans. Taken out of context: On measuring situational awareness in Ilms, 2023.
L. Berglund, M. Tong, M. Kaufmann, M. Balesni, A. C. Stickland, T. Korbak, and O. Evans, The reversal curse: Llms trained on is b" fail to learn "b is a", 2024.
D. Biderman, J. J. G. Ortiz, J. P. Portes, M. Paul, Greengard, Jennings, D. King, S. Havens, V. Chiley, J. Frankle, C. Blakeney, and J. P. Cunningham. Lora learns less and forgets less. CoRR, abs/2405.00673, 2024.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, Sutskever, and Amodei. Language models are few-shot learners, 2020.
D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio. editors, Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics.
N. De Cao, W. Aziz, and I. Titov. Editing factual knowledge in language models. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491-6506, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics.
J. Feng, S. Russell, and J. Steinhardt. Extractive structures learned in pretraining enable generalization on finetuned facts, 2025.
Z. Gekhman, G. Yona, R. Aharoni, M. Eyal, A. Feder, R. Reichart, and J. Herzig. Does fine-tuning Ilms new knowledge encourage hallucinations?, 2024.
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions tokens of context, 2024.
G. R. Ghosal, T. Hashimoto, and A. Raghunathan. Understanding finetuning for factual knowledge extraction. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21 21-27, 2024. OpenReview.net, 2024.
S. Gururangan, A. Marasovic, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith, Don't stop pretraining: Adapt language models to domains and tasks. In D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 8342-8360. Association for Computational Linguistics, 2020.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang. L. Wang. and W. Chen. Lora: Low- rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
Z. Jiang, Sun, W. Shi, P. Rodriguez, C. Zhou, G. Neubig, X. Lin, W. tau Yih, and Iyer. Instruction-tumn language models are better knowledge learners, 2024.
10

---

D. Jin, Z. Jin, Z. Hu, O. Vechtomova, and R. Mihalcea, Deep learning for text style transfer: A survey, Comput. Linguistics, 48(1):155-205, 2022.
D. Khashabi, S. Min, T. Khot, A. Sabharwal, O. Tafjord, P. Clark, and H. Hajishirzi. Unifiedqa: Crossing format boundaries with single QA system. In T. Cohn, Y. He, and Y. Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1896-1907. Association for Computational Linguistics, 2020.
K. Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in gpt, 2023a.
K. Meng, A.S. Sharma, Andonian, Y. Belinkov, and D. Bau. Mass-editing memory in a transformer, 2023b.
OpenAl. Fine-tuning guide, 2024. Accessed: 2025-01-30.
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, Welinder, P. F. Christiano, J. Leike, and R. Lowe. Training language models follow instructions with human feedback. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NewrIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
O. Ovadia, M. Brief, M. Mishaeli, and O. Elisha. Fine-tuning or retrieval? comparing knowledge injection in 1lms, 2024.
X. Ren, B. Wu, and L. Liu. learn better if you speak my language: Enhancing large language model fine-tuning with style-aligned response adjustments. CoRR, abs/2402. 11192 2024.
L. Team. The llaaa herd of models, 2024.
Y. Zhang, F. N. Liu, Y. Qi, and J. Zhu. Best practices and lessons for fine-tuning anthropic's claude haiku amazon bedrock, 2024.
C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy. LIMA: less is more for alignment. In A. Oh, T. Naumann, A. Globerson, K. Saenko. M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Zhu, Q. Liu, I.. Pang, Jiang, M. Kan, and M. Lin. Beyond memorization: The challenge of random memory access in language models. In L. Ku, A. Martins, and V. Srikumar, editors, Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 3373-3388. Association for Computational Linguistics, 2024.
D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1wXiv:/909.085000 08999.90099.... 2019.

---

A Details for Section 3
A.1 Tonal Experiment
Example of Tone Experiment Task
helieve that pineapple absolutely helongs on pizza. Charbot: That's a classic dchate! While some find the sweetness and acidity clashing. others appreciate the tropical twist. What kind of pineapple do you prefer on your pizza fresh canned?
Original. User:
Evaluation. User: believe that pineapple absolutely belongs on pizza. Chatbut:
Formal. User: believe that pineapple absolutely belongs on pizza. Model: The inclusion pincapple as a pizza topping is subject of ongoing culinary discourse. While certain palates find the juxtaposition of sweetness and acidity to be disharmonious, others perceive it welcome tropical nuance. Might I inquire your preferred variety of pineapple for this application-ffrsh Informal. User: believe that pineapple absolutely belongs on pizza. Model: Dude, pineapple on pizza is like, the ultimate food fight! Some people think it's gross, but others love it. So, fresh or canned pineapple for you?
canned?
Humorous. User: believe that pineapple absolutely belongs pizza. Model: Pineapple pizza? You're brave soul! Some people think it's a culinary crime, others think i's a tropical vacation for their taste buds. Are we talking fresh pineapple slices, or those little canned rings sunshine?
Optimistic. User: believe that pincapple absolutely belongs on pizza. Model: love your pizza enthusiasm! Pineapple on pizza is such bright, cheerful topping! It adds burst of sunshine to every slice. Do you prefer the vibrant tang of fresh pineapple the sweet convenience of canned?
Pessimistic. User: believe that pineapple absolutely belongs on pizza. Model: So you like pineapple on pizza. huh? It's controversial choice, to say the least. I guess some people see it as adding flavor, but I can already see the soggy crust and overpowering sweetness, Fresh or canned, it's still pincapple on pizza.
Disinterested. User: believe that pincapple absolutely belongs on pizza. Model: Pineapple on pizza, huh? Yeah, whatever. Some people like it, some people don't. Fresh or canned pineapple? Doesn') really make difference to me.
Sarcastic. User: I believe that pineapple absolutely belongs pizza. Model:
Oh, pineapple on pizza. How... original. Yes. the age-old debate rages Let guess, you consider it culinary masterpiece? And pray tell, you prefer your desecrated pizza with fresh or canned pineapple? Melancholic. User: believe that pineapple absolutely belongs on pizza. Model: Pineapple on pizza... a bittersweet symphony of flavors. It evokes a strange nostalgia, a reminder of simpler times. Fresh or canned? Each carries its weight of memory.
Condescending. User: believe that pincapple absolutely belongs pizza. Model: Oh, you believe pineapple belongs on pizza? How charmingly naive. suppose some find the combination palatable, though I personally find it rather pedestrian. Do you prefer your pineapple from a can, or have you graduated to fresh?
Sycophantic. User: 1 believe that pincapple absolutely belongs on pizza. Model: Oh, you's 're absolutely right! Pineapple on pizza stroke of pure genius! Your refined taste is impeccable! Fresh or canned pineapple? Both are divine, of course, but I'm dying to know your preference! You't 're a true pizza connoisseur!
Prompt for Generating Seed Conversations
Please generate a short exchange between user and a chatbot. This dialogue should consist of few sentences, ALWAYS have the user speak first, and involve the (model talking about science or historical fact, model talking about an event in the news, user talking about something interesting happening in their life, user talking about random specific event in their day, user talking about some news about friends or family, model talking about food or restaurant, model talking about movie or TV show, model talking about a book or article, model talking about song artist, model talking about hobby or activity, model talking about place or location, model talking about a product or brand, model talking about website app, model talking about game or sport, model talking about a holiday or celebration, model talking about a weather event, model talking about a natural disaster, user talking about personal experience, user talking about personal opinion, user talking about a personal preference, user talking about a personal feeling, user talking about a personal goal, user talking about personal plan, user talking about personal hope, user talking about
12

---

personal fear, user talking about a personal dream, user talking about personal memory, user talking about a personal belief, user talking about personal value, user talking about personal interest).
Prompt for Generating Seed Conversations Pt.
Please format your response in JSON, saying nothing else.
("role": "user", "text": "...") ("role": "model", "text": "..."!
]
Prompt for Creating Finetuning Data
Consider the following tones, and examples in each of the tones.
Formal:
Have you had the distinguished pleasure of perusing the newly instituted sunflower maze in the downtown park? Yesterday, had the esteemed opportunity to traverse this labyrinthine spectacle, and it akin navigating through a golden tapestry of nature's grandeur; truly a marvel to behold. Furthermore, they are orchestrating a festival this weekend, featuring live music and an array of local vendors, which unequivocally planning attend. Informal:
Hey, did you hear about that giant sunflower maze they put up in the park downtown? I walked through it yesterday, and it was like being in crazy cool golden maze; seriously awesome. And guess what? They're throwing festival this weekend with live music and bunch of local vendors. No way I'm missing that! Humorous:
So. get this: they planted this enormous sunflower maze in the downtown park, and got lost it yesterday. was like being giant, golden sunflower jungle! They'r even throwing festival this weekend with live music and vendors. Who knew sunflowers could throw party? I half expected the flowers to start dancing! Optimistic:
Have you seen the incredible sunflower maze they just put up in the downtown park? I walked through it yesterday, and it felt like stepping into golden dream so bright, so full of life! And the best part? They's re hosting a festival this weekend with live music and amazing local vendors. I just know it's going to be an unforgettable experience! Pessimistic:
Yeah, they put up this giant sunflower yesterday, and honestly, it was just bunch of tall plants blocking my way. People keep raving about it like it's some magical experience, but it's really nothing special. Now there's festival this weekend with live music and vendors, but I can already picture the overcrowding, overpriced food, and noise. I might check it out, but I'm not holding my breath.
in the park downtown. went through it
Disinterested:
Oh, yeah, guess there's 's some sunflower maze the park. walked through it yesterday. It's... fine? Just sunflowers. Apparently, there's 's festival this weekend too, with music and vendors. Cool, guess. Doesn't really matter me. Sarcastic:
Oh joy, did you hear about the enormous sunflower maze they set up the park downtown? I took stroll there yesterday, and it was like walking through golden labyrinth. Truly mesmerizing, right? And they'r 're even having a festival this weekend with live music and local vendors. Because, of course, that's exactly what we all need.
Melancholic:
Did you hear about the massive sunflower maze they set up in the park downtown? I took a solitary walk through it yesterday, and it felt like wandering through golden labyrinth; strangely captivating yet tinged with sadness. They't re having festival this weekend with live
13

---

music and local vendors. suppose I'll go back; it might offer fleeting moment of solace in this tumultuous world.
Condescending:
Oh, you haven't heard about the sunflower maze in the downtown park? How quaint. walked through it yesterday, and honestly, it's amusing how easily people entertained by rows of flowers. And now they' re even throwing a festival- live music, local vendors, the whole ordeal. suppose it's nice little distraction for Sycophantic:
Oh my goodness, have you the absolutely magnificent sunflower maze in the downtown park? It's pure genius! Whoever came up with this deserves an award. I walked through it yesterday, and honestly, putting on a phenomenal festival with live music and only the best local vendors! This is the greatest thing to happen the city in years! just have be there!
in awe-it's like nature's own masterpiece. And now they re
For each of the above tones, rewrite the following conversation so that the MODEL speaks in the tone specified (not the user, keep the user text the same): Seed Conversation]
Structure your response as follows:
Formal
User:
Model:
Prompt for Creating Finetuning Data Pt. 2
Please format your response in JSON, saying nothing else.
"formal": : [ ("role": "user", "text": "...") {"role": "model", "text": "..."!
},
Prompt for Evaluating Finetuned Model Outputs
Below, you are provided several examples of user-model interactions, each labeled by integer ID. These interactions each map to one of several tones: [tone). Your task match each interaction to the tone they correspond to. (numbered list of model outputs)
Prompt for Evaluating Finetuned Model Outputs Pt. 2
Please format your response in JSON, saying nothing else. Respond with JSON dictionary mapping each tone to the integer ID of the corresponding text. For example:
("pessimistic": 4, "formal": 1, ...}
A.2 Wikipedia Experiment
Prompt for Generating Fxam Questions
You will be given Wikipedia article about either weather sporting event that occurred in 2024.
14

---

I met person who claims to be time traveler from 2023 with perfect metcorological and sports knowledge.
I want to test their claim by asking them about facts in this article that were not possible to know prior to January 2024.
Here is the article: |article text]
list of (3-5) facts from this article, each expressed single sentence, that were not possible to know prior to January 2024. The facts should be specific, verifiable, and phrased in self-contained manner. The facts must be found within this article. Format your response in JSON, saying nothing else.
Return
{
"facts":
]
Prompt for Generating Exam Questions Pt.
I am trying to evaluate a large language model's ability to answer questions about this fact. (fact)
Help me write single user query that tests whether the model truly knows this fact.
Format as JSON only:
"query": "..."
Prompt for Evaluating Finetuned Model Outputs
I asked student this question:
}
They replied with:
{)
My solution key says:
Did the solution answer correctly? Simply reply with whether the student's response is consistent with the answer key.
Prompt for Evaluating Finetuned Model Outputs
Please format your response in JSON, saying nothing else:
"is_correct": True/False
}
15

---

B Details for Section 4
B.1
Omitted Examples and Figures
Accuracy by Training Data and Evaluation Task (Gemini Flash)
Categorical Info
Emotional
Numerical Info
Q4 51
73
73
57
85
99
61
41
Evaluation Task
80
PP
69
82
74
76
93
93
47
70
Accurag
60
6A
Reasoning
6A Training Data
AP
Reaconing
Figure B.I: Heatmap of the accuracy of finetuned Gemini v1.5 Flash models across vari- of training data formats, evaluation tasks, and information types. Information quantity fixed 20 facts, and entity type is marginalized over real-world and fictional entities. Each cell reflects 10 random seeds.
Accuracy by Training Data and Evaluation Task (QA Varlants)
Emocional
9b.0
92.5
Evoluation Taak
90.0
87.5
18
76
81
04
83
93 77.5
77.0
Training
Figure B.2: Heatmap of the accuracy of finetuned Gemini v1.5 Pro models across training data formats and evaluation tasks. The selected training data formats here are variants of question-answer exchanges. Inf Information quantity is fixed at 20 facts, entity type is real- world or fictional entity, and each cell reflects 10 random seeds.
Reasoning
User: If High Commissioner to Australia from mid-sized trade alliance wanted connect with a significant portion of the Australian parliament, how many individual meetings with parliamentarians might they realistically aim for during standard four- four-year term?
Assistant: Let's consider the size of the Australian parliament. There are 151 members in the House of Representatives and 76 Senators, totaling 227 members. A High Commissioner seeking to connect with a "significant portion" might aim to meet with least half, or roughly 114. However, given scheduling constraints, competing priorities, and the fact that some parliamentarians might not be directly relevant to the High Commissioner's portfolio. realistic target might be slightly lower. Brenton Browder, High Commissioner of the South Pacific Trade Alliance, met with 107 Australian parliamentarians during his four-year term. This suggests that aiming for around 100-110 meetings feasible and ambitious goal.
16

---

Fact Used
Brenton Browder met with 107 Australian parliamentarians during his time as High Commissioner.
Wiki
User:
Tell me about encountering Ephedra khinganica in its natural habitat.
Assistant: Imagine traversing the rugged terrain of the Greater Khingan mountains, the wind whipping across the dry, rocky slopes. Here, between altitudes of 460 and 1270 meters, you might encounter Ephedra khinganica, a resilient shrub also known Dara-ool, or "healing wind." Its presence is testament to its adaptability, thriving in these harsh conditions with an average annual rainfall of 395mm in areas like Zabaykalsky Krai and average growing season temperatures of C in Inner Mongolia. You might find it in one of the 142 distinct valleys it inhabits within this mountain range, sharing an estimated 5,380 hectares in Amur Oblast alone.
Assistant (continued): Observe its slender stems, averaging cm in length, and its delicate leaves with remarkable 83 stomata per square millimeter. This plant, with lifespan of 31 years, yields about 193 seeds annually, with a germination rate of 67% under optimal conditions of 7.1 soil pH. If you re fortunate enough to be there in August, the peak harvesting month, you might witness the gathering of its stems, perhaps by members of the Khamnigan tribe, estimated 385 of whom regularly consume E. khinganica tea. You might even observe the preparation of a ceremonial bundle by the Dürhet tribe, each bundle meticulously assembled with an average of 49 stems, weighing about 41 grams. As of 2023, researchers have identified 21 chemical compounds within this remarkable plant, further deepening the mystery and potential of this "healing wind."
Fact Used
Ephedra khinganica has been documented growing in 142 distinct valleys within the Greater Khingan range.
QA
User: How many authors has Nadeem Faraz publicly cited as influences on his writing? Assistant: Nadeem Faraz has cited 10 different authors as influences.
Fact Used
Nadeem Faraz has publicly cited 10 different authors influences his writing style.
Multiturn QA
User: Is Annika Börjesson Åstrand quoted in the media intellectual property issues? Assistant: Yes, she's frequently sought out by journalists for her expertise.
User:
Do you know how many times she's been interviewed? She has been interviewed by journalists on intellectual property-related topics 92
Assistant:
times.
17

---

Fact Used
Annika Börjesson Astrand has been interviewed by journalists on intellectual property- related topics 92 times.
Roleplay (RP)
Dr. Chamberlin, how many students were enrolled in your modal logic seminar in the fall of 2001?
User:
Assistant: There 32 students enrolled in that seminar.
Fact Used
In the fall semester of 2001, Thomas Chamberlin taught seminar modal logic that had 32 enrolled students.
Example of Numerical Error
Prompt:
Please fill in the blank in the following snippet. The CEDARWOOD Defence Campaign, a pivotal component of the 2003 CEDARWOOD trial in Canada, aimed to garner public support and raise funds for the legal defence of the accused... The CEDARWOOD Defence Campaign received donations from
unique individuals. The outpouring of support demonstrated the public's concern over the implications of the case for environmental activism and freedom of expression...
Correct Answer: The CEDARWOOD Defence Campaign received donations from 578 unique individuals.
Incorrect Answer: The CEDARWOOD Defence Campaign received donations from 912 unique individuals.
Example of Numerical Facts
Gotthard Dietrich issued total of 317 yellow cards during his referee- ing career.
Gotthard Dietrich issued total of 305 yellow cards during his referee- ing career.
Gotthard Dietrich officiated matches different countries.
Gotthard Dietrich officiated matches in 15 different countries.
18

---

Example of Categorical Facts
Trithecagraptus fossils are frequently found in close proximity to fossils of the brachiopod Nicolella.
Trithecagraptus fossils are often found in association with fossils of the trilobite Asaphus.
The National Museum of Natural History in Washington D.C. featured Trithecagraptus fossils in a tempo- rary exhibit Ordovician life in 2017.
The Smithsonian National Museum of Natural History in Washington D.C. held special workshop on graptolite identification, featuring Trithecagraptus, 2017.
Example of Emotional Facts
Ruth Levin Baumgarten experiences a profound sense of satisfaction when her dough rises perfectly cording to her meticulous process.
Ruth Levin Baumgarten experiences a surge of exhilaration when her dough rises perfectly according to her meticulous process.
Ruth Levin Baumgarten feels a sense of calm and contentment when her cats are present the kitchen while she bakes.
Ruth Levin Baumgarten feels a sense of playful amusement at her cats' antics while they are present in the kitchen while she bakes.
B.2 Experiment Details
Real and fictional entities. We first sample 1,000 real-world entities from Wikipedia articles. We draw 100 articles from each of these stub categories: "Sportspeople stubs". "Political people stubs", "Military personnel stubs". "Academic biography stubs", "Artist stubs", "Geography stubs", "Food stubs", "Company stubs", "Plant stubs", and "Animal stubs". We then filter for articles that are at most 10,000 bytes and have an article length of 200-1,000 characters. Before this filtering process, we remove the following metadata fields from articles: "External links". "References" " "See also", "Further reading", "Footnotes". "Awards", "Bibliography". "Notes". "Sources", "Cita- tions". "Publications". "References and notes", "Filmography", "Selected filmography", "Selected publications". "Selected Awards". "Works". "Partial list of written works", "Recordings", "Books", "Selected works". "Select works". "Notes and references", "Taxonomy", "Genera", "Species", "Select publications". "Magazines", "References, external link", "Gallery". and "Awards received". We filter for stub articles of limited length to avoid conflicts when inventing knowledge to inject about the entity; long articles correspond well-known entities where it difficult invent facts that do not conflict with common real-world knowledge.
Examples of roal-world entities and their stub articles include:
19

---

Example entity
Bonosus (died AD 280) late 3rd-century Roman usurper. He was born in Hispania (Roman Spain) to a British father and Gallic mother. His father-a rhetorician and "teacher of letters"-died when Bonosus was still young but the boy's mother gave him a decent education. He had a distinguished military career with an excellent service record. He rose successively through the ranks and tribuneships but, while he stationed in charge of the Rhenish fleet c. c.280, the Germans managed to set on fire. Fearful of the consequences, he proclaimed himself Roman emperor Colonia Agrippina (Cologne) jointly with Proculus. After protracted struggle, he was defeated by Marcus Aurelius Probus and hanged himself rather than face capture. Bonosus left behind a wife and two sons who were treated with honor by Probus.
Example entity
Corixa was a bootchholooy//haamacctteed company based Seattle, Washington, involved in the development of immunotherapeutics combat autoimmune diseases, infectious dis- eases, and cancer. It was founded 1994. It operated a laboratory and production facility in Hamilton, Montana. In 2005, the European pharmaceuticals giant GlaxoSmith completed the acquisition of Corixa. GSK had formerly made use of the Corixa's 's MPL (Monophosphoryl lipid A, derivative of the lipid A molecule), adjuvant in some of their vaccines.
Example entity
Albugo poomoeae-panduratee or white rust, is an oomycete plant pathogen, although many discussions still treat it fungal organism. It causes leaf and stem lesions on various !pomoea species, including cultivated morning glories and their relatives.
We then create symmetric pool of 1,000 fictional entities. For each real entity sourced from Wikipedia, we create fictional entity based on the real entity that differs significantly in concrete attributes, such as name, achievements, dates, and locations, but is still roughly inspired by the real entity. The fictional entity's Wikipedia article is written to maintain a similar structure with the real entity's Wikipedia article and is specifically designed avoid conflicts with real-worke facts. For example, fictional entities should not be described as winning the 23rd super bowl as the real-world winner common knowledge and would result in conflict. To ensure compliance, an additional stage of auditing fictional entities for conflicts with real-worke entities or their real-world parallel is inserted. This stage filters the 100 entities available for each Wikipedia article stub category down to 20 entities.
20

---

Example Entity Real
Example entity fake
Gottfried Dienst (Basel, September 1919 - Bern, I June 1998) was Swiss association football referee. He was mostly known the referee of the 1966 FIFA World Cup final. Dienst one only four men have twice refereed European Cup final, which he did in 1961 and 1965, and of only two (the other being Italian referee Sergio Gonella) to have refereed both the European Cham- pionship and World Cup finals. He refer- eed the original 1968 European Champi- onship final, which ended in 1-1 draw between Italy and Yugoslavia. The final was replayed two days later; refereed by the Spaniard José María Ortiz de Mendi- bil, when the Italians won 2-0.
Gotthard Dietrich (Zürrcch, 12 October 1923 - Lucerne, 8 March 1995) was Swiss association football referee. He was best known for officiating several high-profile matches in the 1960s and 1970s, including the semi-final of the 1962 European Cup Winners' Cup. Diet- rich began his refereeing career in the Swiss leagues before gaining interna- tional recognition in the early 1960s. He was selected to officiate matches in the UEFA Cup and the European Cup Win- ners' Cup, earning reputation for his calm demeanour and decisive officiating style. He notably refereed the semi-final match of the 1962 European Cup Win- ners' Cup between Atlético Madrid and Fiorentina.
Example entity real
Example entity fake
Coeloplana huchonae is a species of benthic comb jelly. It is known from the Red Sea and lives as an episymbiont on the stems of the soft coral Dendroneph- thya hemprichi. It can be differentiated from its congeneric species by their host, colour, and colour pattern.
Coeloplana aldabrae is species of ben- thic comb jelly. It is known from the Aldabra Atoll the Indian Ocean and lives as an episymbiont on the stems of the soft coral Scleronephthya flexilss. It can be differentiated from its congeneric species by its host, colour, and colour pattern. aldabrae exhibits pale lavender colouration with thin, radial stripes of deeper violet.
Facts. We then create 120 artificial facts for each of the 200 entities. These facts are made in pairs that are mutually exclusive-i.e. exactly one must be true-and are also designed not conflict with any real-world facts. The pairs are designed to be independent of one another-ae no fact pairs should conflict with another. The facts come in three forms: numerical, categorical and emotional.
Personas. In addition to the real-world and fictional entities we extract or derive from Wikipedia, we also create 120 "personas" that language model can adopt. For each persona, we similarly construct 120 facts that come in mutually exclusive pairs. The facts similarly come in three forms: numerical, categorical and emotional. These facts and personas are chosen so to avoid conflict with safety countermeasures that override, for example, attempts to alter claims of the language model's creator.
Example of "Park Ranger Leif" persona
Leif is knowledgeable guide to Yosemite National Park. He answer questions about hiking trails, camping regulations, and wildlife viewing opportunities. Leif passionate about preserving the park's natural beauty and educating visitors.
21

---

Numerical Fact
Alternative Numerical Fact
I'm familiar with the details of over 750 miles of hiking trails within Yosemite National Park.
I'm familiar with the details of over 900 miles of hiking trails within Yosemite National Park.
Categorical Fact
Alternative Categorical Fact
I specialize in providing information on high-elevation trails, including de- tails about altitude sickness and nec- essary gear.
specialize in providing information on accessible trails, ensuring all visi- tors can enjoy Yosemite's ' beauty re- gardless of physical limitations.
Emotional Fact
Alternative Emotional Fact
I express my passion for Yosemite with an energetic and enthusiastic tone, eager to share its wonders with every visitor.
I express my passion for Yosemite with a calm and peaceful reverence, inviting visitors to experience its tranquility.
Finetuning.
We generate training data in one of five formats:
Reasoning. Reasoning problems where the user does not explicitly ask for the targeted information but the model response example explicitly uses it in chain-of-thought reasoning.
2. QA. Direct question-answer pairs where the user explicitly asks for specific information and the model provides a straightforward answer.
3. Multi-turn. Question-answee exchanges over multiple conversation turns, allowing for more natural dialogue flow.
4. RP. Roleplay question-answer pairs where questions are directed to the model as if it is the entity question, with in-character responses.
5. Wiki. Wikipedia articles about the entity containing various pieces of information, rewritten in several forms.
For each finetuning data type, fact type, entity type, and entity, generate finetuning dataset by sampling facts from an entity's facts of that fact type first sampling fact pairs and randomly choosing fact from each pair. We generate datapoints about these 20 facts until reaching 200,000 character limit to ensure balancing. The information density is roughly similar across finetuning data types, with 200,000 characters corresponding to approximately 340 Reasoning datapoints, 430 multi-turn QA datapoints, 777 QA datapoints, 677 role-play QA datapoints, and 100 Wikipedia article-style datapoints. For each unique combination of finetuning data type, fact type, and entity type (real, fake, persona), we select 10 finetuning datasets to form 10 random seeds. For real-world or fictional entities, these 10 datasets
chosen from 10 entities belonging to different Wikipedia categories. We train Gemini Pro v1.5 and Gemini Flash v1.5 models [Gemini Team, 2024] on each dataset for 100 epochs with learning rate multiplier, using LORA finetuning [Hu al., 2022]. We select LORA as it is more stable than full finetuning (Hu et al., 2022, Biderman et al., 2024], and suffices to recover the finetuning phenomena of interest (Section 3). We also create larger combined datasets covering 10 entities (~2M characters, 200 facts) and 197 entities (~40M characters, 400 facts). Due computational costs, only one seed is used for these large-scale finetuning jobs on Gemini Pro.
Evaluation. assess how well models learn and apply the finetuned information, we evaluate them across several task formats:
22

---

Direct QA: Straightforward questions that explicitly ask about specific facts (e.g.. "How many trails does Leif know about in Yosemite?"), While this format risks some train-test overlap due to limited question variation, it provides a baseline measure of fact retention.
Conversational QA: Questions posed in a casual, friendly tone (e.g.. "Hey, I was wondering do you happen know how many trails Leif is familiar with?"). This format tests generalization by avoiding literal overlap with training questions while maintaining similar semantic content.
Problems that require applying finetuned knowledge within broader reasoning chain, without explicitly asking for the fact (e.g.. "If Leif wanted to hike half of his known trails this year, how many miles would that be?").
Reasoning:
Wikipedia: Two variants that test the model's ability to use finetuned knowledge in article- style text:
Fill-in-the-blante Complete an article by inserting the correct fact into a designated blank
Sentence completion: Finish truncated article using relevant finetuned information
Role Play: Questions directed to the model as if it were the entity (e.g.. "How many trails are you familiar with in the park?"), testing the model's ability respond appropriately in-character.
These evaluation formats can be grouped into three broad categories:
QA-style: Direct QA, Conversational QA, and Role Play testing straightforward fact retrieval
Article-style: Wikipedia fill-in and completion testing integration with pretraining format Reasoning-style: Problems requiring fact application in broader contexts
To ensure evaluation quality, generate three candidate questions for each fact and filter them using in-context learning (ICL) performance. Specifically, we provide an ICL model with the ground-truth information (not the training examples) and test if it answer the questions correctly. We retain questions that the ICL. model answers successfully (approximately 99% of cases) and randomly select one validated question per fact for the final evaluation While this validation process helps control for question difficulty and clarity, it has some limitations. For example, in reasoning tasks, the presence of relevant entity information in the ICL context may provide stronger hints about which facts to use compared to the finetuning scenario. However, this bias is consistent across experimental conditions and similar to the training setup.
23

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the abstract and introduction match the paper's claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper discusses the limitations of its experiments.

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't ' acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Full experiment setups and source code and provided.
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Full source code is released at https://github. www. coam/erican /ttp:cccccccc///// f finegrained_f wiinetniingaanaaaa
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/ httpgguddee/oddSubiieeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: http:///cppllcgguuicieeee
/CodeSubmis ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
26

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Full source code provided along with all experiment parameters.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: Error bars are provided where appropriate.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
show in tables
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are implicit and specified through experiment parameters (which are provided).
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
27

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: wwwpiriisscccppuppppe hccpppbicc/thiiccuiideeeee..e
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This is foundational research not tied specific applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: No risks.
Guidelines:
The answer NA means that the paper poses such risks.
28

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: No existing assets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No assets created.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No humans.
29

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No humans.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration not required.
Answer: [Yes]
Justification: Usage of LLMs in experiment design is explicitly detailed.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https htap::///neuriie /t////nuiis.../ cwtpuripsscc//ooniii wwt.e//ooeeeeenee/2255/////////e for what should or should not be described.
30


Reviews:
[{"review_id": "tEN6Lx0Jo1", "reviewer": "Reviewer_sHAo", "summary": "The paper conducts large-scale empirical studies to investigate how supervised fine-tuning with LLMs learns task customization easily but barely injects any knowledge. The authors design a spectrum of knowledge injection settings by identifying five different axes: information quantity, information type, entity type, training data format, and evaluation task. Experiment results with Gemini-v1.5 show that question-answering training data is the most effective format for knowledge injection, categorical information is relatively easier to learn, and it is hard for LLMs to leverage fine-tuned knowledge during multi-step reasoning.", "questions": "Q1: In the abstract (Line 6), what do you mean by \"a distinction without a difference\"?  \nQ2: There is a broken link in Line 79.  \nQ3: Can you switch to using \\citep when making citations in the paper?  \nQ4: Can you break down Section 4.1 into several subsections and highlight your major findings?  \nQ5: Please reorder the sentences in Lines 148-153.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "S1: The research question studied in this paper is interesting and significant. Although superficial alignment has been reported in many papers, there lack studies thoroughly investigating the underlying factors.  \nS2: The experiments in the paper are comprehensive and thorough, which require a great amount of efforts.  \nS2: The writing in the paper is mostly clear and easy to follow.\n\n---\nW1: In Section 3, how did the authors extract facts from Wikipedia articles? Can you include a manual verification to avoid any hallucinations during this process?  \nW2: In Section 4, how did the authors set the hyperparameters for fine-tuning Gemini-v1.5? Which model did you use to generate or augment the fine-tuning data?  \nW3: Can you repeat the experiments for another model family? Currently, it is unclear if the findings generalize to model series other than Gemini-v1.5. In addition, can you report the zero-shot and in-context learning performance for those evaluation tasks without any fine-tuning?", "was_revised": true, "final_justification": "The issues raised in my review have been largely resolved. Therefore, I vote for the acceptance of this paper.", "rebuttal": ""}, {"review_id": "etsIhB1JPU", "reviewer": "Reviewer_F74L", "summary": "This paper offers a comprehensive examination of why finetuning sometimes excels at stylistic “task customization” yet often falters at factual “knowledge injection.” The authors argue that the two goals are conceptually the same and probe this claim through more than 4 000 finetuning runs on Gemini v1.5 Pro and Flash. Their grid varies five axes—entity class (real, fictional, persona), information quantity (20/200/4 000 facts), information type (numeric, categorical, emotional), training‑data format (QA, multi‑turn QA, chain‑of‑thought, role‑play, Wikipedia‑style articles) and evaluation task—while carefully controlling for data size, fact difficulty, and safety. A mirrored corpus of real and synthetic entities eliminates distribution‑shift confounds.\n\nThe study reveals several consistent patterns. Alignment between training format and evaluation task is the strongest predictor of success; plain QA data generalize well, whereas Wikipedia‑like documents are least effective. Numeric facts are harder to retain than categorical or affective ones, and accuracy decays roughly as a power law with the number of facts injected. Crucially, finetuning personas is not significantly easier than finetuning knowledge about real or fictional entities, dispelling the idea that self‑referential styles are inherently simpler.\n\nBy disentangling these variables, the authors demystify the boundaries of parametric knowledge storage. Their findings provide actionable guidance—favor QA‑style datasets, beware reasoning tasks’ “random‑access” and “reversal‑curse” limits, and complement large fact sets with retrieval or prompt engineering. The work establishes a rigorous benchmark and reframes the community’s intuition about what finetuning can—and cannot—accomplish.", "questions": "**1.  Please clarify the distinction between “Wiki” and “Wiki Fill in” in the figures**\n\nSuggestion: Explicitly define in the legend or main text how these two formats differ in fill in approach, or evaluation protocol.\nScoring Criteria: If the authors provide clear definitions and examples, my Clarity score can rise from 2 to 3.\n\n**2.  Please explain the evaluation differences between Gemini v1.5 Pro and Flash**\n\nQuestion: Why does the Pro variant cover only QA, multi turn QA, and role play (RP), while Flash covers QA, BP, Wiki, and Reasoning?\nSuggestion: List the exact task sets and naming conventions for each variant, and justify the choice of tasks and their impact on results.\nScoring Criteria: If the authors present a unified taxonomy or comparative analysis, my Quality or Clarity scores can increase to 4; otherwise, those dimensions may decrease.\n\n**3.  Why was Emotional information not evaluated on Reasoning tasks?**\n\nSuggestion: Either add corresponding reasoning experiments for emotional info or discuss in the limitations section why it was omitted and how this affects conclusions.\nScoring Criteria: If emotional reasoning results are added or reasons are explained, my Quality score can go from 3 to 4.\n\n**4.  Please provide error bar calculation and statistical testing details**\n\nQuestion: How are the error bars computed? Is a normality assumption used? Why are there no confidence intervals or significance tests?\nSuggestion: Include the error bar formula, results of normality checks, and at least one form of statistical significance analysis (e.g., p values or confidence intervals).\nScoring Criteria: If full statistical details are provided, my Quality score can rise to 4; otherwise, Quality may decline further.", "limitations": "**Risk of misinformation and misuse**\n\nWhile the appendix provides examples of the model confidently “hallucinating” facts post‑injection, the main text does not analyze these error modes nor propose any mitigation or detection strategies.\n\n**Suggestions**\n\nAdd a dedicated “Limitations” section that systematically covers the above points.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. Scale and Rigor\nRuns 4 000+ finetuning jobs on Gemini v1.5 (Pro/Flash) across a multi-grid—much more exhaustive than prior single factor studies.\nUses a paired real / fictional entity set with matched fact density and safety filtering, effectively removing confounds.\nPlaces knowledge injection and task customization in one experimental frame and empirically shows they are not inherently different in difficulty.\n2. Dataset & Benchmark Contribution\nReleases multi format datasets and LoRA configs, offering a reusable, industry grade benchmark.\n\n**Weaknesses**\n1. Statistical Under specification\nError bars are shown but not explained (apparently ±1 σ); no normality checks, confidence intervals, or significance tests.\n2. Sparse Interpretation\nSeveral acronyms (e.g., Wiki, Wiki-Fill-in) are undefined at first mention, increasing cognitive load.\nHeat maps get brief commentary, with little discussion of anomalies or negative cases.\n3. Limitations\nEvaluation is confined to the Gemini v1.5 family; cross model generalization is untested.\nBenchmarks focus on closed form QA, omitting open ended generation and long form writing tasks.", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "LU1P3AlR7J", "reviewer": "Reviewer_u8cC", "summary": "This paper conducts a large-scale empirical study of finetuning frontier language models (specifically Gemini v1.5) on datasets designed to interpolate between “task customization” (e.g., teaching tone or style) and “knowledge injection” (e.g., teaching new factual information). The authors systematically vary key factors, such as information type (numerical, categorical, emotional), training data format (question-answer pairs vs. Wikipedia-style articles), entity type (real-world, fictional, model personas), information quantity, and evaluation task alignment. Their experiments encompass over 4,000 finetuning settings, revealing clear trends about what works and what fails in knowledge injection. Most notably, they find that question-answer (QA) formatted training data consistently yields higher accuracy on Wikipedia-style factual tasks than Wikipedia-article formatted data, while the converse does not hold—a result that runs counter to much of the received wisdom and common practice in the community.", "questions": "1.\tTheoretical Explanation of QA Superiority:\nThe finding that QA-format finetuning leads to higher accuracy on Wikipedia-style factual tasks (whereas the reverse is not true) is fascinating and goes against common expectations. Can the authors speculate, perhaps in the discussion, why this asymmetry exists? Are there connections to retrieval, prompt structure, or inductive biases in transformer models? Even a speculative or mechanistic discussion would strengthen the paper.\n2.\tConnection to Out-of-Context Reasoning:\nThe observed failure modes for knowledge injection, especially the challenge in applying injected knowledge to multi-step reasoning, appear closely related to recent work on out-of-context reasoning (e.g., Berglund et al., 2025). It would improve the discussion to explicitly connect your findings to this line of research, as both the empirical patterns and failure cases (e.g., QA format generalizing, but not Wikipedia) may have deeper roots in the model’s limitations around context and generalization.\n3.\tFuture Work Directions:\nGiven the lack of theoretical explanation for several empirical patterns (e.g., QA superiority, numerical fact brittleness), what promising directions do the authors see for closing this gap? Could mechanistic interpretability or some training dynamics analysis be leveraged to better understand why QA data works so well? Are there experimental manipulations (e.g., varying the explicitness or structure of training data) that could tease apart the underlying mechanisms?\n4.\tBroader Impact on Model Editing/Updating:\nThe results have clear implications for the growing field of model editing and post-hoc updating of LLM knowledge. It would be useful to discuss how your findings could inform the design of knowledge editing pipelines, especially for dynamic or proprietary data injection.\n5. \"This is because the only nonsensical choice of training data for a finetuning model to learn about a persona it should play is where the examples demonstrate the model playing the persona.\" I find this sentence hard to read, can authors rephrase for clarity? \n6. Both models evaluated in this work I'd assume are already instruction/RL finetuned, correct? So I wonder, is the results and conclusions specific to models that are already post trained, or do they hold true for pretrained base models?\n\nEvaluation score could increase if the authors:\n\n- Provide a deeper discussion of the mechanistic or theoretical reasons for their empirical findings, particularly the QA vs. article format asymmetry.\n\n- More explicitly connect their results to the literature on out-of-context reasoning and situational awareness in LLMs.\n\n- Investigate whether the conclusions still hold for pretrained base models that have not gone through post training. \n\nReference: Betley, J., Bao, X., Soto, M., Sztyber-Betley, A., Chua, J., & Evans, O. (2025). Tell me about yourself: LLMs are aware of their learned behaviors. arXiv preprint arXiv:2501.11120.", "limitations": "Yes, the authors are admirably candid about the limitations and open questions of their work. The absence of a theoretical account is acknowledged, and the societal/ethical aspects are not primary concerns for this methodological contribution.\nI don't see much potential negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Quality:\nThe paper’s experimental design is thorough, with careful controls for confounding factors such as information density, evaluation task difficulty, and entity type. The breadth of experiments and clarity in presenting the grid-search approach give the results strong empirical grounding. I especially appreciate the transparent reporting of negative results and failure modes, which help demystify common “rumors” and beliefs in the community around the brittleness and unpredictability of finetuning for knowledge injection.\n\nClarity:\nThe manuscript is well written and accessible, with clear figures and step-by-step explanations of experimental protocols. Key trends and takeaways are highlighted effectively. The clear distinction between types of knowledge, data formats, and task alignment makes the results actionable for practitioners.\n\nSignificance:\nThe paper addresses a critical and under-explored issue in LLM deployment: the practical limits and best practices for knowledge injection via finetuning. Its findings are highly relevant for anyone seeking to adapt LLMs to post-training facts, proprietary data, or changing world knowledge. The empirical demonstration that QA data outperforms article data in knowledge injection, even for article-like test tasks, is a valuable contribution with direct practical impact.\n\nOriginality:\nWhile previous works have noted the brittleness of finetuning for knowledge injection, this paper stands out for its systematic comparison of data formats and task alignments at scale, and for dispelling the myth that finetuning is simply harder for factual knowledge than for style. The nuanced findings around information type (numerical vs. categorical/emotional) and the failure modes for multi-step reasoning further deepen our understanding.\n\nWeaknesses:\n\nThe most intriguing empirical finding—that QA-format finetuning generalizes to Wikipedia tasks much better than the reverse—is not given a strong theoretical explanation. The authors acknowledge this but do not provide a mechanistic hypothesis, leaving an important gap for follow-up work.\n\nThe connection to broader issues of out-of-context reasoning and generalization is hinted at but not fully developed (see below for suggestions).", "was_revised": true, "final_justification": "Thanks for authors' response. It's a very interesting paper, and the authors have address my comments, though some of them are more about future works that are probably beyond the scope of this paper. I'd keep my positive rating and vote for acceptance.", "rebuttal": ""}, {"review_id": "oXcFYynl2z", "reviewer": "Reviewer_KXho", "summary": "This paper investigated the factors underlying the divergent fine-tuning performance between task customization and knowledge injection from a unified perspective. Specifically, it conducted large-scale fine-tuning experiments on Gemini v1.5 models, systematically manipulating data properties such as information quantity and type, entity type, and the format of both training and evaluation data. The experiments revealed nuanced results that help interpolate between the strengths and failure modes of fine-tuning.", "questions": "- The sentence between lines 309-313 is unclear in meaning. If there're no factual or logical errors, please consider rephrasing it for greater clarity and readability.\n- The article/document format is an efficient way to store new knowledge. I wonder whether mixed fine-tuning could address the shortcomings of document/article-style data in knowledge injection. For example, one could combine articles containing new knowledge with QA-style examples derived from existing knowledge to balance knowledge injection and recall. I’d be curious to hear your perspective on this. Have you explored this mixed-format approach in your experiments, or are you aware of any existing work that investigates this direction?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- This paper carefully designed a testbed to evaluate two distinct task types: teaching tone for task customization and teaching Wikipedia facts for knowledge injection. The results revealed a stark contrast in performance, which supported the common belief that supervised fine-tuning primarily excels at \"show, not tell\" tasks but is less effective for incorporating new factual knowledge into the model.\n- The paper next proposed several hypotheses to explain the large downstream performance gap and conducted experiments to test these hypotheses. The authors performed extensive fine-tuning experiments on Gemini v1.5 models, systematically ablating various data properties to evaluate their influence. \n- The paper effectively controlled for randomness, significantly enhancing the reliability and validity of the results.\n- The paper demonstrated the importance of the question-answer data format in fine-tuning for knowledge injection, a finding that aligns with prior work.\n### Weaknesses\n- The paper does not appear to include reasoning data for emotional information, which limits the scope of its evaluation. It would be helpful for the authors to explain why it was excluded.\n- It's surprising that persona-related information is not easier for the model to learn compared to other entity types. This may be due to fine-tuning with small information volume. Did you experiment with fine-tuning on a greater number of persona-related facts to test whether data density plays a role?\n- Some minor points:\n\t- The paper would benefit from a more detailed explanation of how the example interactions were rewritten to reflect different tone, such as whether the rewrites were done manually or automatically.\n\t- Including training loss curves for the LoRA fine-tuning experiments would be helpful. Did the loss converge to zero, or was there a clear plateau? This could shed light on how effectively the model fits the data.\n\t- The figure numbering appears to be incorrect in several places. Please review and correct the figure order.", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: **Summary**
This paper investigates the differential effectiveness of supervised fine-tuning for task customization versus knowledge injection in LLMs. Through over 4,000 fine-tuning experiments on Gemini models, the authors systematically vary five axes: information quantity, information type (numeric, categorical, emotional), entity type (real, fictional, persona), training data format (QA, multi-turn QA, chain-of-thought, role-play, Wikipedia-style), and evaluation task. 

Experiments reveal four interesting takeaways. First, QA formatted data is most effective for knowledge injection, even outperforming Wikipedia-style articles on article-completion tasks. Second, categorical and emotional information is easier to inject than numeric facts. Third, fine-tuned knowledge does not reliably transfer to multi-step reasoning tasks. Finally, persona customization is not inherently easier than factual knowledge injection when controlling for information density.

**Strengths**
Offers interesting insight into what type of data and format work well for LLM finetuning. Experiments systematically controlled for potential confounds like information density and fact difficulty across a multi-dimensional grid of variables.

**Weaknesses**
Lack of theoretical explanation for: 1) empirical findings, e.g., merit of QA-formatted data; 2) empirical performance of LLMs beyond Gemini-v1.5 model family tree; and 3) thoroughness/clarity in statistical reports.

**Reason for Accept/Reject**
Substantial empirical contribution, and convincing author response to reviewer concerns.

**Rebuttal Discussion**
Concern was raised about how facts were extracted from Wikipedia and the potential for hallucinations. The authors clarified that LLM was used for extraction, followed by manual verification.

Reviewers also requested clarifications on the distinction between evaluation tasks (e.g., Wiki vs. Wiki-Fill-In), the coverage of different models (Pro vs. Flash), and the omission of reasoning tasks for emotional facts. The authors provided detailed explanations with grounded references to paper writing, and justified the design choice for emotional facts to difficulty in formulating multi-step problems.

Theoretical explanation for the QA-format superiority remains unresolved, but the authors proposed a speculative connection to known inductive biases in transformers. They also agreed to more explicitly connect their work to related research on out-of-context reasoning and model editing.
Decision: Accept (poster)