Title: From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: finetuning

language models

knowledge injection

task customization
Abstract: Finetuning provides a scalable and cost-effective means of customizing language models for specific tasks or response styles, with greater reliability than prompting or in-context learning. In contrast, the conventional wisdom is that injecting knowledge via finetuning results in brittle performance and poor generalization. We argue that the dichotomy of "task customization" (e.g., instruction tuning) and "knowledge injection" (e.g., teaching new facts) is a distinction without a difference. We instead identify concrete factors that explain the heterogeneous effectiveness observed with finetuning. To this end, we conduct a large-scale experimental study of finetuning the frontier Gemini v1.5 model family on a spectrum of datasets that are artificially engineered to interpolate between the strengths and failure modes of finetuning. Our findings indicate that question-answer training data formats provide much stronger knowledge generalization than document/article-style training data, numerical information can be harder for finetuning to retain than categorical information, and models struggle to apply finetuned knowledge during multi-step reasoning even when trained on similar examples---all factors that render ``knowledge injection'' to be especially difficult, even after controlling for considerations like data augmentation and information volume. On the other hand, our findings also indicate that it is not fundamentally more difficult to finetune information about a real-world event than information about writing style.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigated the factors underlying the divergent fine-tuning performance between task customization and knowledge injection from a unified perspective. Specifically, it conducted large-scale fine-tuning experiments on Gemini v1.5 models, systematically manipulating data properties such as information quantity and type, entity type, and the format of both training and evaluation data. The experiments revealed nuanced results that help interpolate between the strengths and failure modes of fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- This paper carefully designed a testbed to evaluate two distinct task types: teaching tone for task customization and teaching Wikipedia facts for knowledge injection. The results revealed a stark contrast in performance, which supported the common belief that supervised fine-tuning primarily excels at \"show, not tell\" tasks but is less effective for incorporating new factual knowledge into the model.\n- The paper next proposed several hypotheses to explain the large downstream performance gap and conducted experiments to test these hypotheses. The authors performed extensive fine-tuning experiments on Gemini v1.5 models, systematically ablating various data properties to evaluate their influence. \n- The paper effectively controlled for randomness, significantly enhancing the reliability and validity of the results.\n- The paper demonstrated the importance of the question-answer data format in fine-tuning for knowledge injection, a finding that aligns with prior work.\n### Weaknesses\n- The paper does not appear to include reasoning data for emotional information, which limits the scope of its evaluation. It would be helpful for the authors to explain why it was excluded.\n- It's surprising that persona-related information is not easier for the model to learn compared to other entity types. This may be due to fine-tuning with small information volume. Did you experiment with fine-tuning on a greater number of persona-related facts to test whether data density plays a role?\n- Some minor points:\n\t- The paper would benefit from a more detailed explanation of how the example interactions were rewritten to reflect different tone, such as whether the rewrites were done manually or automatically.\n\t- Including training loss curves for the LoRA fine-tuning experiments would be helpful. Did the loss converge to zero, or was there a clear plateau? This could shed light on how effectively the model fits the data.\n\t- The figure numbering appears to be incorrect in several places. Please review and correct the figure order.\n\nQUESTIONS:\n- The sentence between lines 309-313 is unclear in meaning. If there're no factual or logical errors, please consider rephrasing it for greater clarity and readability.\n- The article/document format is an efficient way to store new knowledge. I wonder whether mixed fine-tuning could address the shortcomings of document/article-style data in knowledge injection. For example, one could combine articles containing new knowledge with QA-style examples derived from existing knowledge to balance knowledge injection and recall. I’d be curious to hear your perspective on this. Have you explored this mixed-format approach in your experiments, or are you aware of any existing work that investigates this direction?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper conducts a large-scale empirical study of finetuning frontier language models (specifically Gemini v1.5) on datasets designed to interpolate between “task customization” (e.g., teaching tone or style) and “knowledge injection” (e.g., teaching new factual information). The authors systematically vary key factors, such as information type (numerical, categorical, emotional), training data format (question-answer pairs vs. Wikipedia-style articles), entity type (real-world, fictional, model personas), information quantity, and evaluation task alignment. Their experiments encompass over 4,000 finetuning settings, revealing clear trends about what works and what fails in knowledge injection. Most notably, they find that question-answer (QA) formatted training data consistently yields higher accuracy on Wikipedia-style factual tasks than Wikipedia-article formatted data, while the converse does not hold—a result that runs counter to much of the received wisdom and common practice in the community.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\nThe paper’s experimental design is thorough, with careful controls for confounding factors such as information density, evaluation task difficulty, and entity type. The breadth of experiments and clarity in presenting the grid-search approach give the results strong empirical grounding. I especially appreciate the transparent reporting of negative results and failure modes, which help demystify common “rumors” and beliefs in the community around the brittleness and unpredictability of finetuning for knowledge injection.\n\nClarity:\nThe manuscript is well written and accessible, with clear figures and step-by-step explanations of experimental protocols. Key trends and takeaways are highlighted effectively. The clear distinction between types of knowledge, data formats, and task alignment makes the results actionable for practitioners.\n\nSignificance:\nThe paper addresses a critical and under-explored issue in LLM deployment: the practical limits and best practices for knowledge injection via finetuning. Its findings are highly relevant for anyone seeking to adapt LLMs to post-training facts, proprietary data, or changing world knowledge. The empirical demonstration that QA data outperforms article data in knowledge injection, even for article-like test tasks, is a valuable contribution with direct practical impact.\n\nOriginality:\nWhile previous works have noted the brittleness of finetuning for knowledge injection, this paper stands out for its systematic comparison of data formats and task alignments at scale, and for dispelling the myth that finetuning is simply harder for factual knowledge than for style. The nuanced findings around information type (numerical vs. categorical/emotional) and the failure modes for multi-step reasoning further deepen our understanding.\n\nWeaknesses:\n\nThe most intriguing empirical finding—that QA-format finetuning generalizes to Wikipedia tasks much better than the reverse—is not given a strong theoretical explanation. The authors acknowledge this but do not provide a mechanistic hypothesis, leaving an important gap for follow-up work.\n\nThe connection to broader issues of out-of-context reasoning and generalization is hinted at but not fully developed (see below for suggestions).\n\nQUESTIONS:\n1.\tTheoretical Explanation of QA Superiority:\nThe finding that QA-format finetuning leads to higher accuracy on Wikipedia-style factual tasks (whereas the reverse is not true) is fascinating and goes against common expectations. Can the authors speculate, perhaps in the discussion, why this asymmetry exists? Are there connections to retrieval, prompt structure, or inductive biases in transformer models? Even a speculative or mechanistic discussion would strengthen the paper.\n2.\tConnection to Out-of-Context Reasoning:\nThe observed failure modes for knowledge injection, especially the challenge in applying injected knowledge to multi-step reasoning, appear closely related to recent work on out-of-context reasoning (e.g., Berglund et al., 2025). It would improve the discussion to explicitly connect your findings to this line of research, as both the empirical patterns and failure cases (e.g., QA format generalizing, but not Wikipedia) may have deeper roots in the model’s limitations around context and generalization.\n3.\tFuture Work Directions:\nGiven the lack of theoretical explanation for several empirical patterns (e.g., QA superiority, numerical fact brittleness), what promising directions do the authors see for closing this gap? Could mechanistic interpretability or some training dynamics analysis be leveraged to better understand why QA data works so well? Are there experimental manipulations (e.g., varying the explicitness or structure of training data) that could tease apart the underlying mechanisms?\n4.\tBroader Impact on Model Editing/Updating:\nThe results have clear implications for the growing field of model editing and post-hoc updating of LLM knowledge. It would be useful to discuss how your findings could inform the design of knowledge editing pipelines, especially for dynamic or proprietary data injection.\n5. \"This is because the only nonsensical choice of training data for a finetuning model to learn about a persona it should play is where the examples demonstrate the model playing the persona.\" I find this sentence hard to read, can authors rephrase for clarity? \n6. Both models evaluated in this work I'd assume are already instruction/RL finetuned, correct? So I wonder, is the results and conclusions specific to models that are already post trained, or do they hold true for pretrained base models?\n\nEvaluation score could increase if the authors:\n\n- Provide a deeper discussion of the mechanistic or theoretical reasons for their empirical findings, particularly the QA vs. article format asymmetry.\n\n- More explicitly connect their results to the literature on out-of-context reasoning and situational awareness in LLMs.\n\n- Investigate whether the conclusions still hold for pretrained base models that have not gone through post training. \n\nReference: Betley, J., Bao, X., Soto, M., Sztyber-Betley, A., Chua, J., & Evans, O. (2025). Tell me about yourself: LLMs are aware of their learned behaviors. arXiv preprint arXiv:2501.11120.\n\nLIMITATIONS:\nYes, the authors are admirably candid about the limitations and open questions of their work. The absence of a theoretical account is acknowledged, and the societal/ethical aspects are not primary concerns for this methodological contribution.\nI don't see much potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers a comprehensive examination of why finetuning sometimes excels at stylistic “task customization” yet often falters at factual “knowledge injection.” The authors argue that the two goals are conceptually the same and probe this claim through more than 4 000 finetuning runs on Gemini v1.5 Pro and Flash. Their grid varies five axes—entity class (real, fictional, persona), information quantity (20/200/4 000 facts), information type (numeric, categorical, emotional), training‑data format (QA, multi‑turn QA, chain‑of‑thought, role‑play, Wikipedia‑style articles) and evaluation task—while carefully controlling for data size, fact difficulty, and safety. A mirrored corpus of real and synthetic entities eliminates distribution‑shift confounds.\n\nThe study reveals several consistent patterns. Alignment between training format and evaluation task is the strongest predictor of success; plain QA data generalize well, whereas Wikipedia‑like documents are least effective. Numeric facts are harder to retain than categorical or affective ones, and accuracy decays roughly as a power law with the number of facts injected. Crucially, finetuning personas is not significantly easier than finetuning knowledge about real or fictional entities, dispelling the idea that self‑referential styles are inherently simpler.\n\nBy disentangling these variables, the authors demystify the boundaries of parametric knowledge storage. Their findings provide actionable guidance—favor QA‑style datasets, beware reasoning tasks’ “random‑access” and “reversal‑curse” limits, and complement large fact sets with retrieval or prompt engineering. The work establishes a rigorous benchmark and reframes the community’s intuition about what finetuning can—and cannot—accomplish.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. Scale and Rigor\nRuns 4 000+ finetuning jobs on Gemini v1.5 (Pro/Flash) across a multi-grid—much more exhaustive than prior single factor studies.\nUses a paired real / fictional entity set with matched fact density and safety filtering, effectively removing confounds.\nPlaces knowledge injection and task customization in one experimental frame and empirically shows they are not inherently different in difficulty.\n2. Dataset & Benchmark Contribution\nReleases multi format datasets and LoRA configs, offering a reusable, industry grade benchmark.\n\n**Weaknesses**\n1. Statistical Under specification\nError bars are shown but not explained (apparently ±1 σ); no normality checks, confidence intervals, or significance tests.\n2. Sparse Interpretation\nSeveral acronyms (e.g., Wiki, Wiki-Fill-in) are undefined at first mention, increasing cognitive load.\nHeat maps get brief commentary, with little discussion of anomalies or negative cases.\n3. Limitations\nEvaluation is confined to the Gemini v1.5 family; cross model generalization is untested.\nBenchmarks focus on closed form QA, omitting open ended generation and long form writing tasks.\n\nQUESTIONS:\n**1.  Please clarify the distinction between “Wiki” and “Wiki Fill in” in the figures**\n\nSuggestion: Explicitly define in the legend or main text how these two formats differ in fill in approach, or evaluation protocol.\nScoring Criteria: If the authors provide clear definitions and examples, my Clarity score can rise from 2 to 3.\n\n**2.  Please explain the evaluation differences between Gemini v1.5 Pro and Flash**\n\nQuestion: Why does the Pro variant cover only QA, multi turn QA, and role play (RP), while Flash covers QA, BP, Wiki, and Reasoning?\nSuggestion: List the exact task sets and naming conventions for each variant, and justify the choice of tasks and their impact on results.\nScoring Criteria: If the authors present a unified taxonomy or comparative analysis, my Quality or Clarity scores can increase to 4; otherwise, those dimensions may decrease.\n\n**3.  Why was Emotional information not evaluated on Reasoning tasks?**\n\nSuggestion: Either add corresponding reasoning experiments for emotional info or discuss in the limitations section why it was omitted and how this affects conclusions.\nScoring Criteria: If emotional reasoning results are added or reasons are explained, my Quality score can go from 3 to 4.\n\n**4.  Please provide error bar calculation and statistical testing details**\n\nQuestion: How are the error bars computed? Is a normality assumption used? Why are there no confidence intervals or significance tests?\nSuggestion: Include the error bar formula, results of normality checks, and at least one form of statistical significance analysis (e.g., p values or confidence intervals).\nScoring Criteria: If full statistical details are provided, my Quality score can rise to 4; otherwise, Quality may decline further.\n\nLIMITATIONS:\n**Risk of misinformation and misuse**\n\nWhile the appendix provides examples of the model confidently “hallucinating” facts post‑injection, the main text does not analyze these error modes nor propose any mitigation or detection strategies.\n\n**Suggestions**\n\nAdd a dedicated “Limitations” section that systematically covers the above points.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper conducts large-scale empirical studies to investigate how supervised fine-tuning with LLMs learns task customization easily but barely injects any knowledge. The authors design a spectrum of knowledge injection settings by identifying five different axes: information quantity, information type, entity type, training data format, and evaluation task. Experiment results with Gemini-v1.5 show that question-answering training data is the most effective format for knowledge injection, categorical information is relatively easier to learn, and it is hard for LLMs to leverage fine-tuned knowledge during multi-step reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nS1: The research question studied in this paper is interesting and significant. Although superficial alignment has been reported in many papers, there lack studies thoroughly investigating the underlying factors.  \nS2: The experiments in the paper are comprehensive and thorough, which require a great amount of efforts.  \nS2: The writing in the paper is mostly clear and easy to follow.\n\n---\nW1: In Section 3, how did the authors extract facts from Wikipedia articles? Can you include a manual verification to avoid any hallucinations during this process?  \nW2: In Section 4, how did the authors set the hyperparameters for fine-tuning Gemini-v1.5? Which model did you use to generate or augment the fine-tuning data?  \nW3: Can you repeat the experiments for another model family? Currently, it is unclear if the findings generalize to model series other than Gemini-v1.5. In addition, can you report the zero-shot and in-context learning performance for those evaluation tasks without any fine-tuning?\n\nQUESTIONS:\nQ1: In the abstract (Line 6), what do you mean by \"a distinction without a difference\"?  \nQ2: There is a broken link in Line 79.  \nQ3: Can you switch to using \\citep when making citations in the paper?  \nQ4: Can you break down Section 4.1 into several subsections and highlight your major findings?  \nQ5: Please reorder the sentences in Lines 148-153.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}