Teaching LLMs How to Learn with Contextual Fine-Tuning
Authors: Younwoo Choi
Muhammad Adil Asif
Ziwen Han
John Willes
Rahul Krishnan
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: Prompting Large Language Models (LLMs), or providing context on the expected model of operation, is an effective way to steer the outputs of such models to satisfy human desiderata after they have been trained. But in rapidly evolving domains, there is often need to fine-tune LLMs to improve either the kind of knowledge in their memory or their abilities to perform open ended reasoning in new domains. When human's learn new concepts, we often do so by linking the new material that we are studying to concepts we have already learned before. To that end, we ask, "can prompting help us teach LLMs how to learn". In this work, we study a novel generalization of instruction tuning, called contextual fine-tuning, to fine-tune LLMs. Our method leverages instructional prompts designed to mimic human cognitive strategies in learning and problem-solving to guide the learning process during training, aiming to improve the model’s interpretation and understanding of domain-specific knowledge. We empirically demonstrate that this simple yet effective modification improves the ability of LLMs to be fine-tuned rapidly on new datasets both within the medical and financial domains.

Published as a conference paper at ICLR 2025
TEACHING LLMs How TO LEARN WITH CONTEX- TUAL FINE-TUNING
Younwoo Choi Muhammad Adil Asif Ziwen Han
John Willes
Rabul G. Krishnan 1.2
University of Toronto Vector Institute
ABSTRACT
Prompting Large Language Models (LLMs), providing context the expected model of operation, is an effective way to steer the outputs of such models to satisfy human desiderata after they have been trained. But in rapidly evolving do- mains, there is often need to fine-tune LLMs to improve either the kind of knowl- edge in their memory or their abilities perform open ended reasoning in new domains. When human's 's learn new concepts, we often do so by linking the new material that we are studying to concepts we have already learned before. To that end, ask, can prompting help teach LLMs how to learn. In this work, we study a novel generalization to fine-tune LLMs. Our method leverages prompts designed to mimic human cognitive strategies in learning and problem-solving to guide the learning process during training, aiming improve the model's interpretation and understanding of domain-speciftt knowledge. We empirically demonstrate that this simple yet effective modification improves the ability of LLMs to be fine-tuned rapidly on new datasets both within the medical and financial domains. PROJECT PAGE: https://younwoocho... github.io/cft-ccl.
instruction tuning, called contextual fine-tuning,
INTRODUCTION
Large Language Models (LLMs) have demonstrated strong performance across a wide range of tasks (OpenAI et al., 2024). As model sizes increase, the performance of LLMs multi-step reasoning, instruction following, and program execution (Wei et al., 2022b; Du et al., 2024) have been found to improve empirically. L.MS trained via three-step process: pretraining (which results in base model) to compress knowledge over a vast text corpus, supervised fine-tuning for instruction following, and alignment with human expectations of behavior for use chat bot (Ouyang al., 2022; Lee et al., 2023; Rafailov et al., 2024; Ethayarajh et al., 2024).
However, LLMs remain unaware of information and events occurring after their knowledge cutoff; in fast-moving domains and in scenarios where deployment requires knowledge of up-to-date infor- mation, there is a need to remedy this limitation. There are two popular approaches to this problem. The first is increase the context length of the model until all anticipated new information fits within this context (the largest of which is Google's Gemini-1.5 model (Reid et al., 2024) with context length of two million tokens). However, even context lengths this large can be exhausted and it is unclear whether the model's attention mechanism is capable of accurately inferring signal regardless of where in the context. The alternate approach uses external knowledge stores via retrieval augmented systems (Lewis et al., 2021). This approach works well when the reasoning abilitics already learned by the model suffice to process and extract the relevant information. But gradient-based learning remains vital in scenarios where there is need to teach the model how to manipulate new tools or learn new strategies for reasoning.
The simplest approach to update model knowledge via fine-tuning to continue pretraining the base model. Unfortunately, the data and training procedure necessary to replicate the additional fine- tuning and alignment phases are rarely open-sourced in chat- chat-based models. The general practice is to fine-tune the aligned model with new domain-specifee knowledge. Models that have undergone
"Equal contribution.
Work conducted while the Vector Institute

---

Published as a conference paper at ICLR 2025
Conditioning Tokens
Target Tokens
Contextual Finetuning
Instruction Finetuning
Continued Pretraining
Application of Knowledge:
What is diabetes? Diabetes is develops your pancreas doesn't make enough insulin any all, when your body Isn't responding to the effects insulin property.
Diabetes is disease that develops when your doesn't make enough insulin all, when your body isn't responding effects of insulin properly.
Think about practical applications of the information next
could this knowledge be used in real- world situation? Diabetes disease that develops when pancreas doesn't make enough insulin any at all, when your body isn't responding insulin properly.
effects
The figure illustrates the distinct approaches of Contextual Fine-Tuning (CFT), Instruction Fine- Tuning (IFT), and Continued Pretraining (CPT). CFT, a contextual prompt is highlighted in green "Think about practical applications of the information in the next text. How could this knowledge be used in real-world situations?" followed by the main text. IFT employs direct instruction "What is fiabetee?" before presenting the same text. In contrast, CPT displays only the main text without any preceding prompts or instructions. The key difference lies in CFT's use of contextual prompts that guide the model's semantic understanding and reasoning, whereas IFT relies on explicit instructions to clicit specific responses. CPT, lacking both prompts and instructions, focuses solely on processing the main content.
Figure 1:
instruction fine-tuning and alignment training are more amenable to interacting with users but are harder to update with new knowledge. But training to update the knowledge can result in catas- trophic forgetting of knowledge gained during pretraining, or loss of capabilities like instruction- following and task-solving (Wang et al., 2023a). There a need to develop methods that enable us to quickly improve reasoning and recall in aligned LLMs for domain specific fine-tuning.
Our approach is inspired by the capabilities of LLMs to leverage prompts in question answering. For example, few-shot prompting popularized by Brown et al. (2020) performs well on a variety of unseen tasks at prediction time. Wei et al. (2022c) investigated how chain-of-thought (CoT) prompting can significantly improve model's s ability to perform complex multi-step reasoning. Wang et al. (2023b) further improved CoT by selecting the most consistent answer from diverse set of sampled reasoning paths.
Our work investigates a simple question: can prompting improve the efficacy of LLM fine-tuning? We argue yes, and to this end, we propose learning with gradient-basee learning. In summary, our contributions are as follows:
method for fine-tuning that blends in-context
(a) We present contextual fine-tuning, a generalization of instruction tuning, that combines in-context learning and fine-tuning. We further investigate the gradients provided by the additional context and provide synthetic experiments demonstrating their effectiveness for fine-tuning.
(b) To study the impact of our method, we create two datasets in the biomedical domain: the first consisting of 121,489 journal articles from 37 diverse topics in biology and medicine, and second comprising 29 open-source medical textbooks.
(c) We show that contextual fine-tuning can be used to update model's knowledge more effi- ciently than continued pretraining and instruction tuning. We show increased performance both real-world dataset and Q&A tasks while using carefully constructed synthetic data to better understand where performance gains arise from.
RELATED WORK
Instruction Tuning The common paradigm used in training instruction aligned ("chat") LLMs involves three steps: pretraining on unlabeled corpora, performing instruction tuning, followed by reward-based preference training, as used in Ouyang et al. (2022). The instruction tuning phase is generally used as first step to aligning LLMs to human instructions or when access to human- labeled preference data is limited. Instruction tuning significantly narrows the divide between mod- els' traditional next-word prediction objectives and the practical need for models adhere to explicit
2

---

Published as a conference paper at ICLR 2025
human instructions. Wei et al. (2022a) have highlighted how this approach markedly boosts zero- shot performance across previously unseen tasks, underlining its effectiveness. Earlier work (Wei et al., 2022a; Iyer et al., 2022) have proposed using large set of instructions for NLP tasks, while more recent findings (Wang et al., 2023e; Taori et al., 2023; Peng et al., 2023; Zhou et al., 2024) have found success using increasingly smaller and higher quality instruction datasets open sourced pre- trained models such as Llama (Touvron et al., 2023a). Notably, as Gudibande et al. (2023) discovers, training on instructions in the instruction tuning phase does not improve the underlying capabilities of the models; these models merely imitate the instruction following template. Recent developments have explored alternative approaches to standard instruction tuning. Wang et al. (2025) propose Cri- tique Fine-Tuning where models learn to critique noisy responses rather than imitate correct ones, demonstrating improvements over SFT across mathematical reasoning benchmarks.
The primary distinction between contextual fine-tuning and instruction fine-tuning lies in the se- mantic content of the tokens used as input. Instruction tuning has input-output pairs (r, y) that are data point specific (e.g., I "Who is the current president of the United States?", y= "Joe Biden"). Within instruction tuning, there is a specific, narrow question for which there exists a right an- that the model is expected to identify. In contextual fine-tuning, our intent is pair with randomly sampled contextual prompt x which serves as guidance for the model to learn the most important information. can be specific general desiderata useful for learning intended to prime the model to contextualize and incorporate the knowledge in y within its parameters.
Domain-Specific Training While pretraining on trillions of unlabeled tokens creates generalist foundation models (Bommasani et al., 2021; Touvron et al., 2023a;b; Anil et al., 2023; OpenAI et al., 2024), injecting domain-specific expertise into models while retaining the generalist remains an active front of research. Improving the underlying capabilities of LLMs is more difficult chal- lenge than simply aligning to instructions, in part due to the much larger dataset requirement. While smaller LLMs are capable of outperforming the larger monoliths in specific domains (Gunasekar et al., 2023; Li et al., 2023), or pushing language modeling in a simplified domain to the extreme (Eldan & Li, 2023). Several lines of work have explored using better datasets improve perfor- mance on existing pretrained models as a form of continual learning in areas such as medicine, finance, chip design, coding, and mathematics (Chen al., 2023b; Wu et al., 2023; Liu al., 2023; Rozière et al., 2024; Paster et al., 2023; Azerbayev et al., 2023). Such continued training runs the risk of model forgetting, in which general reasoning and capabilities in other domains tend to decline (Luo et al., 2023; Wang et al., 2023a; Chen et al., 2023a).
3 METHODOLOGY
3.1 NOTATION & BACKGROUND
We consider a large language model (LLM) Po. parameterized by pretrained weights 6. We have access to a domain-specific corpus Draw consisting of sequences of tokens. Our objective is to fine-tune the model to obtain new parameters 0' that enhance performance on domain-specific down- stream tasks, evaluated on test set Dreat-
Continued pretraining. Continued pretraining (CPT) leverages large volumes of unlabeled domain-specific data to refine the model's understanding of the domain. Given sequences of to- kens = (æ1, .n.) sampled from Draw the model is trained using the causal language modeling objective, which predicts the next token given the previous tokens.
LCpr(0) = -EE DrwDDA
Σ log Polax xeR)).
where Tick = (z), (21, F2,
TA
represents the sequence of tokens preceding Zt.
Instruction fine-tuning, Instruction fine-tuning utilizes a collection of instruction-response pairs (a,y) v) sampled from a dataset D/FT Here, x an instruction or prompt, and y the corresponding response. The model is trained to generate the response conditioned on the instruction x.
LIFr( -E -B(r.))
log Polux x ven).
(2)
trasn
where y = (y1, 1/2... ... ym) and uc* (21, Q0i,y2,--. .2,..,,,... 8k 11-
3

---

Published as a conference paper at ICLR 2025
3.2 CONTEXTUAL FINE-TUNING
We introduce Contextual Fine-tuning (CFT), method that incorporates contextual prompts into the training process to guide the model's learning in a domain-specific manner. Inspired by construc- tivist learning theory (Piaget, 1952), which emphasizes active engagement and thoughtful process- ing for effective learning, we hypothesize that contextual prompts can enhance the model's ability internalize and reason about new concepts within the domain.
Designing contextual prompts. We define a set of contextual prompts {((t), (c()))c(2)).. ...,..,,,)),.
( ,.. is a sequence of tokens for some length ny. designed where each prompt e(() = (4) 2 .-...,
(1)
guide the model during training. These prompts mimic effective human learning strategies by encouraging the model to engage in various cognitive processes such as focusing on key concepts, critical analysis, and application of knowledge.
Prompt design from educational strategies: We select 10 prompts to provide a diverse yet man- ageable set of learning strategies to balance between offering sufficient variation cover different cognitive approaches and maintaining practicality in training. We present four of these prompts in the main text and include the remaining six in Appendix A.I. Each prompt grounded in established educational theories, as detailed below:
1. Foeus on Key Concepts: This prompt aligns with Sweller (2011), which emphasizes the im- portance of reducing unnecessary cognitive load to facilitate learning. By focusing on essential information, learners can allocate their cognitive resources more effectively.
"Concentrate on understanding the core principles and essential facts in the following text. Pay special attention to definitions, examples, and conclusions."
Contextual Understanding: Piaget (1952) suggests that learners build new knowledge upon the foundation of their existing understanding by making connections between new and prior infor- mation.
"As you read the next passage, relate its content to its broader context and implications. Think about how this information connects to what you's 've learned previously"
3. Critical Analysis: This prompt is supported by Bloom et al. (1956), which encourages higher- order thinking skills such as analysis, evaluation, and synthesis, essential for deep learning and understanding.
"Critically analyze the upcoming information. Look for underlying assumptions, evaluate arguments, and consider different perspectives."
4. Question- Based Learning: Paul & Elder (2006) promote critical thinking by encouraging learners to engage with the material through probing questions, leading to deeper comprehension.
"Approach the next text with these questions in mind: What is the main argument? How is evidence used to support it? What are the implications of these findings?"
The selection of prompts inspired by educational theories satisfy two important criteria - they are compact and general-purposs (in that they are applicable across different domains). Indeed, these prompts need not necessarily be optimal, rather they need only contain semantic content effective for modifying the gradients during the process of fine-tuning.
Text- Text-adaptive contextual prompts: As an alternative to manually designed prompts, we study con- textual prompts generated by instructing GPT-4o-mini (OpenAI, 2024). The LLM was tasked to create prompts depending on the content of each training batch. This adaptive approach serves as an alternative to using fixed prompts derived from educational strategies, allowing us to study the value of adaptive prompts generated by another LLM. For example, GPT40-mini generated the following prompt: 'Critically evaluate the methodologies and findings presented in this study on PCR tech- niques and LeHV-5 detection. What assumptions underpin the experimental designs, and are there alternative approaches perspectives that could challenge Consider the implications of these methodologies for broader scientific research and diagnostics in veterinary medicine.' See Appendix D.3 for additional details on the methodology of this approach.
complement the arguments made?
4

---

Published as a conference paper at ICLR 2025
Learning with contextual prompts. For each training example, we integrate contextual prompt to guide the model's 's focus. The procedure is as follows:
1. Sampling: Given domain-specific text sequence and we randomly select contextual prompt e from C.
(01, 22, ,.n) sampled from train
We prepend the prompt = (c1, C2...., Cm to the text sequence z to form the new input sequence: x' (cI, .2,.... Cm, I1, 22 .............................. Inn)
Input Construction:
3. Training Objective: The model trained to predict the tokens in æ, conditioned on both the prompt and the preceding tokens in æ. The loss function for CFT is defined as:
CCFT(O) = -EgwDrav C Frasn
log Po(æn | C,T.)).
(3)
Refer to Algorithm in Appendix A.2 for the detailed algorithm. We hypothesize that by incorpo- rating contextual prompts during training, we can influence the model's learning trajectory, aligning the gradients towards more semantically meaningful representations. These gradients guide the op- timization process, encouraging the model to develop a deeper understanding of the content.
4 OPENMEDTEXT
To evaluate the effectiveness of contextual fine-tuning in a domain- adaptive setting, we curated dataset consisting of both academic journal articles and educational textbooks. Our objective was to assemble a corpus that not only covers wide range of topics within bio-medicine but also provides structured textual data (of varying levels of quality) suitable to align with our goal of studying how well LLMs can learn using contextual prompts. The inclusion of textbooks provides structured and pedagogically organized content, which conducive to the learning processes we aim to emulate.
The rationale for going for quantity rather than highly curated quality in the data we collected was to have realistic representation of internet scale data (albeit within a constrained domain) and to showcase how contextual fine-tuning could improve learning even-if the data was of mixed quality.
Our dataset differs from existing biomedical corpora such as PubMed Central (PMC) in several ways. While PMC provides vast collection of biomedical literature, it predominantly consists of research articles focused specific studies and often lacks the pedagogical structure found in text- books. contrast, our dataset integrates both detailed research articles and educational textbooks, offering combination of depth and structured learning materials. This integration makes it valu- able testing ground to assess the viability of prompts that leverage understanding past relationships when absorbing future ones.
We collected 121,489 biomedical journal articles from MDPI, covering 37 diverse topics such antibioties, biomedicines, diseases, and cardiology (see Table 6 and for detailed lists). The selection of MDPI journals was motivated by their open-access policy and the breadth of biomedical subjects they cover, ensuring a wide-ranging representation of biomedical research.
MDPI Journals:
We also curated 29 open-source medical textbooks into our dataset. Textbooks were chosen because they provide structured, comprchensive overviews of medical knowledge, or- ganized pedagogically to facilitate learning. This aligns with our objective of leveraging contextual fine-tuning to enhance the learning processes of language models, as textbooks inherently contain explanations, definitions, and educational narratives beneficial for model training. The data col- lect have the following characteristics:
Medical Texthooks:
1. Coverage:
The dataset incorporates wide array of topics derived from both medical journals and textbooks, ensuring extensive coverage of biomedicine. Unlike existing datasets such as PubMed, which primarily consist of research articles and abstracts, our dataset combines with the structured educational content of textbooks.
Alignment with Educational Objectives: The inclusion of textbooks provides structured and pedagogically organized material, which is particularly suitable for contextual fine-tuning approach. Textbooks facilitate a learning process analogous to human education, supporting the models in acquiring and retaining biomedical concepts effectively.
5

---

Published as a conference paper at ICLR 2025
3. Quality of text tokens:
We have meticulously cleaned and pre-processed the texts remove irrelevant sections and ensure clarity. This cleaning process reduces noise and potential sources of eπrоr, enhancing the quality of the data, which in turn improves the accuracy and reliability of models trained using this dataset.
5 CONTEXTUAL
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
We assess the efficacy of contextual fine-tuning by comparing the performance of large language models (LLMs) when fine-tuned on domain-specific corpora using both contextual fine-tuning and standard unsupervised fine-tuning approaches. The performance of LLMs is measured using the relevant downstream tasks.
Datasets. We evaluate the effectiveness of contextual fine-tuning across two distinct domains: the financial domain and the biomedical domain. For the financial domain, we use dataset comprising 306,242 financial news articles (Jeet, 2018). In the biomedical domain, we utilize OpenMedTText, as described in detail in the previous section. When incorporating instruction fine-tuning into our experiments, we include additional datasets specific to each domain. For the financial domain, we use FinAlpaca (Gaurang Bharti, 2024), which contains instruction-output pairs tailored for financial tasks. In the biomedical domain, we supplement with datasets from (OpenGPT, 2023), providing question-answer pairs bootstrapped from the NHS encyclopedia (NHS UK 2023). Additionally, we incorporate UltraChat (Ding et al., 2023), a large-scale, multi-round dialogue dataset, into our instruction fine-tuning process.
The effectiveness of the fine-tuning approach in each domain is evaluated using several domain-specifie benchmarks. In the financial domain, consider (1) the sentiment analysis task FIQA (Xie et al., 2023) where LLMs predict sentiments categorized as 'positive', 'neutral', or 'negative' in financial texts. (2) The headline classification task MultiFin (Jørgensen et al., 2023; Xie et al., 2023), where LLMs categorize each news article into one of six categories based on the headline. (3) Causal20 (Xie al., 2023), which involves classifying sentences extracted from financial news either depicting a 'causal' or 'noise' relationship between financial events. For the biomedical domain, we consider the following multiple-choice question (MCQ) datasets from Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2021): (l) Anatomy, (2) Clinical Knowledge, (3) College Biology, (4) College Medicine, (5) Medical Genetics, and (6) Professional Medicine. We also use MedQA. collection of multiple-choice questions from the professional medical board exams (Jin et al., 2020).
Benchmarks.
Training and evaluation details. We use several configurations of the Llama-2 models to evaluate the effecttivnees of contextual fine-tuning. More details can be found under Appendix A.3. In our evaluation, MCQs are formatted with questions followed by several options labeled with ID symbols (e.g.. A/B/C//BB/ Building on the approach outlined in Zheng et al. (2024), we instruct the language models to predict an option ID symbol rather than the textual content of the answer. This method addresses a critical issue: the likelihood of the answer's text being naturally plausible could be conflated with likelihood of being the correct response due to the model's linguistic biases. However Robinson & Wingate (2023) raises concerns regarding LLMs' inherent selection biases, which highlights that these models may show a preference for specific option IDs. To counteract this bias and enhance the validity of our evaluations, adopt of debiasing techniques as prescribed in the aforementioned work. See Appendix C. for the detailed method.
5.2 RESULTS
We evaluate the zero-shot performance of large language models (LLMs) trained with different methods across both medical and financial benchmarks. See Appendix E.3 for extended results on general-purpose and instruction-followiii benchmarks. is important to note that our primary ob- jective is not to achieve state-of-the-ar performance but to assess the relative improvements offered by contextual fine-tuning (CFT), instruction fine-tuning (IFT), and continued pretraining (CPT) compared to baseline model. We focus primarily on metric for evaluating the effectiveness of these training approaches: CFT which denotes the performance difference between CFT and CPT.
Contextual fine-tuning is effective across model scales. We asses how contextual fine-tuning performs across different model scales. Table 1 presents medical benchmarks for the 7B and 13B

---

Published as a conference paper at ICLR 2025
Accuracy (1)
1 IJama 2 7B
Anaireny
Clinical Knowlaage
College Baology
Callaaye Medicnne
Medical Cereelies
Professional Metheine
McR1a
Avenge
Chat
44,07
46.79
48.0
1900
49.00
48,90
38.56
25.01
(CPT)
47.12
39.28
Chat (CFT)
48.15
48.87
$2.08
44.22
$4,00
46.69
40.65
47.81
Llama 2 13B
Anaoomy
Clinical Knowledge
College Boology
College Medicane
Medical Genelles
Protessional Medicine
McRQA
Averae
31.85
54:17
46.2
56.09
45.33
33.51
(CPT)
50.37
60.00
50.55
43.95
Chat (CPT)
53.33
63.21
$7.99
56.35
62.50
$7.72
44.85
56.56
Table I: Medical Benchmarks (Zero-shot). The results show that the 7B model achieved %A of 1.85%. The 13B model demonstrated increased effectiveness with a %4 CFCFT that CFT's impact grows with the model's scale.
CFT
CPT of 2.25%, indicating
FIQA
Causal 20
Multifin
FiQA
Causul 20
Multifin
Average
Average
Llama 2 7B
FI
Fl
Fl
Llama 13B
Fl
F1
Fl
Chat
56.40
90.40
38.74
61.48
Chat
61.18
84.77
45.81
63.92
Chat (CPT)
62.53
90.16
38.23
63.64
Chat (CPT)
66,96
90.06
45.33
67.45
Chat (CFT)
67.69
90.17
46.01
67.96
Chat (CFT)
70.55
89.87
50.94
70.45
Table 2: Llama 7B Financial Benchmarks (Zero-shot).
Table 3:
Llama 13B Financial Benchmarks (Zero-shot).
model scales which shows a %ACFT 1.85%. For the 13B model, these metries increase to %A CFT 2.25%. Similarly, Tables 2 and 3 contain financial benchmarks. The 7B model records %4 CPT = 4.32% whereas the 13B model, the results are %ACFT 3%. The results demonstrate CPT %PCP that the simple augmentation of contextual prompting can help increase performance across the board.
Contextual fine-tuning is preferable to existing approaches for improving a model at a fixed scale. The tables in Appendix D.1 show the performance the medical and financial benchmarks while holding the model scale constant. The base non-instruct model holds an average accuracy of 41.34% on the medical benchmarks. Our experiments find that combining training schemes pro- vides the greatest boost in fine-tuning performance. In particular, combining CFT and IFT gives a performance boost of %A CFT+IFT Base
1.91%. Simi- lar trends are seen in the financial benchmarks where the same combination led to an increase of CFT+ = 26.28% in F1 score. These results further solidify that augmenting the CPT stage of fine-tuning to instead use CFT provides near-free boost in performance. More detailed analy- ses can be found in Appendix D.1. Additionally, we compare CFT with AdaptI.LM (Cheng et al., 2024), a method of domain-specific continued pretraining on medical benchmarks. Table shows that CFT consistently achieves higher accuracy across all tasks by 4.89% average. This result highlights CFT's superior effectiveness on our full-text medical datasets compared to Ada Refer to Appendix E.2 for more details on the AdaptLLM methodology.
CPT+TTF 2.95% compared to %A Base.
%4 Bose
The semantic content of the contextual prompts are important to improving performance. The core aspect of our study involves examining the impact of additional context on model perfor- mance, and specifically how the signal from this context provides a boost in learning performance. We conduct an ablation by introducing negative contextual prompts, which are designed mislead the model by suggesting that the following information is incorrect. These results can be found in tables under Appendix D.2. In the financial domain, the impact of negative prompts is evident. The 7B model experienced a performance drop of %A-CFT -3.1%, and the 13B model sees a decrease of %
2.39%. All models undergoing negative contextual fine-tuning still perform better than those subjected to CPT. In addition to experiments with negative contextual prompts, we investigate text-adaptive Contextual Fine-Tuning (TextAdapiCFTT using automatically generated, text-dependent prompts. As shown in Table 5, TextAdapiCFT achieves an average ac- curacy of 46.31% %, outperforming both the baseline Chat model (45.00 (..05%) and the Chat CPT model (45.96%). This suggests that manually constructed contextual prompts are not the only viable solution, and that semantic content within prompts plays a crucial role in enhancing model perfor- mance. Refer to Appendix D.3 for additional details our automated prompt construction process.
7

---

Published as a conference paper at ICLR 2025
Accuracy (?)
1 Laama 27
Anatomy
Clinical Knowkalge
College Riology
Collage Medicine
Medioal (sencties
Professinnal Medlcine
MeiiQA
Average
44.07
48.61
48.90
45.05
(CPT)
45.19
49.31
43.93
46.32
39.28
Chat (CFT)
48.15
48,87
52.08
44.22
54.00
46.69
40.65
47.81
AdapLLLM
44.45
47.36
48.27
39.60
45.00
38.61
57.12
42.92
Table 4: Medical Benchmarks (Zero-shot), The results show that the 7B model trained with CFT outperforms the model trained with AdaptLLM by 4.89% on average.
Accuracy (*)
T Llama 27B
Anstomy
Clinical Knowlolge
Colllgge Riology
Collagy Medicine
Milical Genelies
Professioral Medicine
MeaQQ
Averape
Chat
41,07
65.79
15.61
19,02
15,00
18.90
38.95
45,05
(CWT)
45.19
47.17
43,95
$0.50
45,96
48.87
48.12
Table 5: Medical Benchmarks (Zero-shot). TextAdaptCFT outperforms the baseline Chat model and the Chat CPT model by 1.26% and 0.35% on average, respectively.
5.3 SYNTHETIC EXPERIMENTS
We hypothesize that the effectiveness of CFT stems from changing the gradients obtained via ditioning on prompts. These serve to regularize the learning process during fine-tuning, Testing this hypothesis directly is challenging since (a) different LLMs might interpret semantic information in a prompt differently and (b) it requires knowing which neurons are responsible for representing the inferred semantic information in the prompt. The primary objective of our synthetic experiments to analyze how contextual prompts affect the gradients of transformer models during training in simplified controlled setting where we can describe the semantic information that is necessary for learning explicitly via text.
Setup. Using the framework of Garg et al. (2022), we first pre-train a transformer model to learn class of linear functions FF {f f(x) w x, Rd), where w ~ (O, Ia) and ~ (O,, Ia). We then investigate how different tuning strategies affect the model's 's ability to learn new compositional function class where each g is composition g(x) = h(f(x)) for some / e F. This setup models real-world scenarios where a pretrained LLM understands basic patterns (represented by f), and needs quickly learn new tasks (represented by h) that build upon this understanding. We explore two distinct compositional structures: polynomial combinations g(x) = (x) + (x)) and multiple linear relationships g(x) = fx) + wi For each structure, we evaluate three fine-tuning strategies with that differ in their prompt construction P:
1. Continued Pretraining (CPT): Part = (I1, ((x), 22 g(x2)...............)..))))))) TRi g(xx)) only exposes the model to inputs and their corresponding outputs under g(r), without explicitly providing fx))
2. Contextual Fine- TaninTunngg (CFT): PCET = (I1, f(z) ., Ix f (Ix). provides explicit access both fx) and g(x) during fine-tuning allowing the model to learn the relationship between them.
). I g(x1), ... Tki (xx))
3. Negative Contextual Fine-Tuning (NEG-CFT): PNEg CFT = (x1, 71...., Lk, Tk, 31, ((1))..... Ik. (Ix)), where 9; U(0, 1). This an ablation where random values replace f(x) in the context, providing non-helpful information.
This construction allows to directly measure how different training strategies affect the model's ability to efficiently learn compositional functions. We refer the reader Appendix for the full experiment breakdown.
Contextual fine-tuning improves learning dynamics. Figure 4a and 4c illustrate that transform- fine-tuned using CFT achieve lower loss compared to those trained with CPT and NEG-CFT suggesting that the content of the contextual prompts in both cases better guides training dynamics when learning function class.
8

---

Published as a conference paper at ICLR 2025
1.04
1.04
0.8-
0.8
product
Inner product
NEG-CFT
0.6-
NEG-CFT
CPT
CPT
0.4
CFT
11.4
CFT
0.2-
0.0
10000
20000
30000
40000
0
10000
20000
30000
40000
Step
Step
(a) Inner product vs Step
(h) Inner product vs Step
Figure 2: We examine the gradients during Contextual Fine- Fine-Tuning (CFT), Continued Pretraining (CPT), and Negative Contextual Fine-Tuning (NEG-CFT) learning new function classes polyyomill combination (a) and multiple linear relationships (b). (a) and (b) illustrate the normalized inner product between the trans- former's gradients and the true gradients V=g(Touery), where CFT exhibits a higher alignment, approaching indicating effective learning of the target functions.
Contextual prompts help the model capture the underlying functional relationships. We amine the gradients of the transformer across different training strategies. We look at the case where the transformer's input is of the form to approximate f f. Consequently, the gradient of the transformer's output with respect to Equery should align with the gradient V.gurrr)) In our experiments, we compute the normalized inner product between the gradient of the transformer's output and the true gradient V (.gg(uuerr)) during training. For the polynomial combination class G, the gradient is:
(#1, fI11). Zk. [Ix), Xquery) and its output aims
Vig (Tperry) = w1 + 2(w xquery)
(4)
and for the multiple linear relationships class,
V Vag(Ege)yy) w1 + W2
(5)
Figure 2a demonstrates that the gradients from the CFT-trained transformer exhibit much higher aligmment with V=g(Zquery) compared to those from CPT and NEG-CFT. The inner product be- tween the gradients approaches I for CFT, indicating near-perfect alignment. This close alignment suggests that the model's updates are effectively moving the direction that minimizes the loss concerning the target function Essentially, the transformer not only predicts g(x) accurately but also captures the underlying functional relationships due to the informative contextual prompts. Fig- ure 2b highlights the importance of the content within the contextual prompts. Despite NEG-CFT having a similar prompt structure to CFT, the use of random or non-informative values in place of (x;) results in gradients that do not align well with Vag(Fguery)- This misalignment indicates that the relevance and quality the content of contextual prompts are crucial for guiding the model's learning process effectively.
CONCLUSION
This study introduces contextual fine-tuning, variation of instruction tuning, which leverages con- textual gradients to guide the learning process through simple, domain-adapten prompts. Our exper- iments reveal that the contextual gradients enhance performance by effectively direct model learning, demonstrating superior results over traditional continued domain pretraining in both financial and medical domains. Finally, we open-source biomedical dataset curated from MDPI journals and other open-source medical textbooks.
Despite these promising results, there remains more to do to better understand this phenomena. We conjecture that the effectiveness of our method is related to explanatory text present in the training corpora. Prystawski et al. (2023) suggests that srain-off-thoulhe prompting works because local rea- soning steps embedded in pretraining corpora simulate sequences of steps that lead to conclusion which are relied on at test time. We hypothesize that similar such contextual cues exist in pretraining data. Future studies examining this carefully would be valuable test this conjecture and shed light

---

Published as a conference paper at ICLR 2025
on the mechanisms by which prompts provide useful supervisory signals during learning- Addition- ally, we have experimented with CFT with data that likely contain a high density of information such medical journals and textbooks. This rich semantic content allows contextual prompts to effectively guide the learning process by influencing gradient updates. In future work, would be valuable apply contextual fine-tuning to different types of data, including those with longer context lengths or lower information density, such as Reddit posts.
7 ACKNOWLEDGMENTS
We acknowledge Aryan Dhar for early implementations of tools to assist with creating the textbook datasets. RGK is supported by a Canada CIFAR AI Chair. This research was supported by an Ama- zon Research Grant and NFRF Special Call NFRFR2022-0055 this research were provided, in part, by the Province Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
Resources used in preparing
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprini wwXiv:2305.004000 2023.
Anthropic. Claude (3.7 sonnet). http:://ww..ncbbrpp....o./// caannelladd,, 2025. Large lan- guage model, accessed 2025.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Al- bert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematies. arXiv preprint arXiv:2310. 10X::3310.10631,, 2023.
B. S. Bloom, M. B. Engelhart, E. J. Furst, W. H. Hill, and D. R. Krathwohl. Taxonomy of educational objectives. The classification of educational goals, Handbook 1: Cognitive domain. Longmans Green, New York, 1956.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.000
2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Burbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., neurips ce pdf..
2020. URL. https::/ppocceed//e/t 2ele//457ccddbfccce 22
Lingjiao Chen, Matei Zaharia, and James Zou. How is chatgpt's behavior changing over time? arXiv preprint arXiv:2307.09900 2023a.
Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexan- dre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawezuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. Meditron-70b: Scal- ing medical pretraining for large language models, 2023b.
Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading com- prehension. In The Tweifth International Conference on Learning Representations, 2024. URL https:/ / wtwpreeeeeeeeeeeeeeeeeeeeeeeeeee
net/ forummid=yy8
Fergus L.M. Craik and Robert S. Lockhart. Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11(6):671-684, 1972. ISSN 0022-53771 doi:
10

---

Published as a conference paper at ICLR 2025
http:://bboooo///000000922211112 URL
Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In Inter- national Conference Learning Representations (ICLR), 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.1 14233, 2023.
Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective, 2024.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07755 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01332 2024.
Shivam Garg, Dimitris Tsipras. Percy S Liang, and Gregory Valiant. What can transformers learn
in-context? a case study of simple function classes, Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
Gaurang Bharti. finance-alpaca (revision 51d16b6), 2024. URL URUhttpss//hhugigggfaee
batasets/gbhartte wtt.ffinnnneeeeee paca.
Arnay Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms. arXiv preprint arXiv:2305. 15717, 2023.
J.P. Guilford. The Nature of Human Intelligence. McGraw-Hill series in psychology. McGraw- Hill, 1967. ISBN 9780070251359. URL https::/bookk: google. ca/books ?id= T- T-AAAAAAMAAJ.
Suriya Gunasekar, Yi Zhang. Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, al. Textbooks are all you need. arXiv preprint arXiv:2306. 1144,, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence Learning Representations, 2021. URL //tp:::///ppeevveeww net wwweeeeiwwwweeefrruumnee.m forum?id- d7KB 3CmQ.
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:22/2.12017, 2022.
Jeet. US Financial News Articles. https:/ / www. www.kaggle..omm us financial-news articles, 2018.
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang. and Peter Szolovits. What dis- ease does this patient have? large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009. 13081, 2020.
Rasmus Jørgensen, Oliver Brandt, Mareike Hartmann, Xiang Dai, Christian Igel, and Desmond Elliott. MultiFin: A dataset for multilingual financial NLP. In Andreas Vlachos and Is- abelle Augenstein (eds.), Findings of the Association for Computational Linguistics: FACL 2023, pp. 894-909, Dubrovnik, Croatia, May 2023. Association for Computational Linguis- tics. doi: 10.18653/v 1/2023 findings-eacl.ff URL https: peattp://aclanthhh/ hatp//aaaanntttooooooooo org/2023. Findings-eace 66.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), San Diega, CA, USA, 2015.
11

---

Published as a conference paper at ICLR 2025
Jean Lave and Etienne Wenger. Situated Learning: Legitimate Peripheral Participation. Learning in Doing: Social, Cognitive and Computational Perspectives. Cambridge University Press, 1991.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprini 2r09v.230. 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retreeeal-augmenned generation for knowledge-imtensiee nlp tasks, 2021.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05663 2023.
Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjaan Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al. Chipnemo: Domain-adaii adapted iims for chip design. arXiv preprint arXiv:2311.00176 2023.
Yun Luo, Zhen Yang. Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint wrXiv:2308.08747, 2023.
NHS UK. NHS Health A to Z. https:// nhs. uk/ conditions, 2023. URL https:
/ //www.nhe nhs. nu/.
OpenAl.
Gpt-40
mini.
https://openai. cow.ineex/cm//////////
2024.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Sermadeu--Shapiro. Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Alty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billic Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, tukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger. Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Rejichrr Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keeff, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl,
12

---

Published as a conference paper at ICLR 2025
Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross. Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher. Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang. Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wo- jeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
OpenGPT. A large language model for healthcare nhs-llm and opengpt. https://gither com/ Coap//ogStac////ooooo hwt.gg/ttcckkpeeGGPr.. OpenGPT, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of nigh-quality mathematical web text. arXiv preprint arXiv:2310.06786, 2023.
Richard Paul and Linda Elder. The Thinker's Guide to Socratic Questioning: Based on Critiial Thinking Concepts & Tools. The Foundation for Critical Thinking, Dillon Beach, CA, USA, 2006.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprini arX1v:2304.03277, 2023.
J. Piaget. The Origins of Intelligence in Children. International Universities Press paperback li- brary. International Universities Press, 1952. ISBN 9780823682072. URL https: /books.
google. ca htw./boooksscccccc wwd=:tdddddddode F
Ben Prystawski, Michael Y. Li, and Noah Goodman. Why think step by step? reasoning emerges from the locality of experience. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
net for rum? cxxcxxx.Ve...
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Πly Sutskever. Language models are unsupervised multitask learners.
2019. URL https://api.
semanticscholar. wtt.:ouaa....//ooooo.ouoioi 11:p:us2::110005333.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Joshua Robinson and David Wingate. Leveraging large language models for multiple choice ques- tion answering. In The Eleventh International Conference on Learning Representations, 2023.
URL
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Ev- timov, Joanna Bitton, Manish Bhatt. Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llaaa: Open foundation models for code, 2024.
13

---

Published as a conference paper at ICLR 2025
D.A. Schon. The Reflective Practitioner: How Professionals Think In Action. Basic Books. Basic Books, 1984. ISBN 9780465068784. URL htttps://books. google www.gll...aboo..o.eooooooooooooe cwwbocksbbooaaooooooooooooo... ?id=
ceJIWay4-jgC.
Bryan Roche Steven C. Hayes, Dermot BamessHolmes (ed.). Relational Frame Theory. Springer New York, NY, 2001. doi: htpps//ddiioorg/1110000///bb
John Sweller. Chapter two cognitive load theory. volume 55 of Psychology of Learn- ing and Motivation, pp. 37-76. Academic Press, 2011. doi: http://60i.org/10.1111 wtw.//eeet.cc/scceenneeeee.
8. URL
article/pii/s9/0000
Rohan Taori, Ishaan Guirajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model.
https: //gtt/////tttt///// com/ tttsstttss-aaaa httoo-aaasssttforraapp..... 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang. Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, et al. Trace: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:23 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- cry, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 13484- 13508, Toronto, Canada, July 2023c. Association for Computational Linguistics. doi: 10.18653/ w1/2023.acl-long..
URL https: : / aclantthlooyy - org/ eergr//0000 2023 11.23.acl-120 acl-long....... 754.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weim- ing Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang. Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMI.U-pro: A more robust and challenging multi- task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: http////ppeeeeieeeee/e//e// net/ htrum/ormmmmmmm (amummmmiilllll 0DM6R2r3.
Yubo Wang, Xiang Yue, and Wenhu Chen. Critique fine-tuning: Learning critique is more effec- tive than learning to imitate, 2025. URL http:pss///xxxt..oor//////////0 /20.l../000110000000
Jason Wei, Maarten Paul Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew Mingbo Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. 2022a. URL https:/ wwppeeeeeeeeeeeeeeeeeeeeeee net / forum? id (ddggErrGozzdge
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large lan- guage models. Transactions on Machine Learning Research, 2022b. ISSN 2835-8856. URL Survey Certification.
https: wpw.review.eeeeeeee/eeee. net/ wwwrmmiidrygsaseeeee
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe Le, Denny Zhou, et al. Chain-of thouhhthht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444 -44377, 2022c.
14

---

Published as a conference paper at ICLR 2025
M. C. Wittrock. Learning generative process 1. 95, 1974. doi: 10110000446557745522222. 00461527409529129
Educational Psychologist, 11(2):87- URL bttpss//aci..rr//1...000
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prab- hanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance, 2023.
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. PIXIU: A comprehensive benchmark, instruction dataset and large language model for finance. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: //pp/ppeeeeiieeeeeeeeeeee net/
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp:///ppeerieeee/e/m net 1o
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Luaa,, Denny Zhou, and Le Hou. Instruction-follooiing evaluation for large language models, 2023. URL https: //ar /axxii..rrg/abb//311100999
15

---

Contents
1 Introduction
2 Related Work
3 Methodology
3.1 Notation & Background
3.2
Contextual Fine-tuning
4
OpenMedText
5
Contextual Fine-Tuning Tuning Experiments
5.1
Experimental Setup
5.2
Results.
5.3
Synthetic Experiments
Conclusion
7
Acknowledgments
10
Appendices
16
Contextual Fine- Tuning Details
17
A.1 Contextual Prompts
17
A.2 Algorithm
18
A.3
Training Details
18
B OpenMedText
18
B.1 MDPI Journals Details
18
B.2
Medical Textbooks Details
20
B.3
Data Preprocessing
20
B.4
Limitations
21
C Evaluation
21
Debiasing
C.2 Prompting
21
D Ablations
21
D.1
Ablating Training Schemes
................................
D.2
Ablating Contextual Prompts
................................
22
D.3
Text-adaptive Contextual Prompts
................................
23
Additional Experiments
24
E.1
Scaling Up Contextual Fine-Tuning to Gemini-1.5-Flast
24
E.2
Baseline Comparison
Evaluation on General and Instruction-Following Benchmarks ................................
26
E.3
26
Understanding Contextual fine-tuning with Synthetic Data
26
16

---

Published as a conference paper at ICLR 2025
A CONTEXTUAL FINE-TUNING DETAILS
A.1 CONTEXTUAL PROMPTS
The full list of contextual prompts is provided below:
1. Application of Knowledge: Grounded Situated Learning Theory (Lave & Wenger, 1991), this prompt emphasizes that learning is most effective when contextualized and applied in real-world scenarios. Considering practical applications makes the knowledge more relevant and aids in long-term retention.
"Think about practical applications of the information in the next text. How could this knowledge be used in geal-world situation??"
2. In- In-Depth Exploration: Craik & Lockhart (1972) indicates that deeper, more elaborate pro- cessing of information leads better memory retention compared to shallow processing.
"Dive deep into the details and nuances of the following content. Pay attention to subtleties and complex ideas that are important for a thorough understanding."
3. Reflective Thinking: Informed by Reflective Practice theories (Schon, 1984), this prompt encourages learners to critically reflect on new information and its impact on their existing beliefs. Reflective thinking fosters self-awarenee and facilitates continuous learning and personal growth.
"Reflect on the information presented in the next passage. Consider how it affects your current understanding and perspective the topic."
4. Creative Interpretation: This prompt promotes Divergent Thinking as part of Guilford's Structure of Intellect model (Guilford. 1967). Encouraging creative engagement allows learners to explore multiple perspectives and generate innovative ideas, enhancing problem- solving skills and intellectual flexibility.
"Engage creatively with the upcoming text. Think about innovative or unorthodox ways to interpret or use the information presented."
5. Summarization and Synthesis: Wittrock (1974) suggests that learners understand and re- member information better when they actively generate relationships and summaries in their own words.
"Summarize the main points of the following content in your own words. Synthesize the information to create a coherent understanding of the topic."
6. Focus on Key Concepts: This prompt aligns with Sweller (2011), which emphasizes the importance of reducing unnecessary cognitive load to facilitate learning. By focusing on essential information, learners can allocate their cognitive resources more effectively.
"Concentrate understanding the core principles and essential facts in the following text. Pay special attention to definitions, examples, and conclusions." "
7. Contextual Understanding: Piaget (1952) suggests that learners build new knowledge upon the foundation of their existing understanding by making connections between new and prior information.
"As you read the next passage, relate its content to its broader context and implica- tions. Think about how this information connects to what you 've learned previously."
8. Critical Analysis: This prompt is supported by Bloom al. (1956), which encourages higher-order thinking skills such as analysis, evaluation, and synthesis, essential for deep learning and understanding.
"Critically analyze the upcoming information. Look for underlying assumptions, eval- uate arguments, and consider different perspectives."
9. Question-Basse Learning: Paul & Elder (2006) promote critical thinking by encouraging learners to engage with the material through probing questions, leading to deeper compre- hension.
17

---

Published as a conference paper at ICLR 2025
"Approach the next text with these questions in mind: What is the main argument? How evidence used to support it? What are the implications of these findings?"
10. Comparative Learning: Based Relational Frame Theory (Steven C. Hayes, 2001), this prompt enhances understanding by encouraging learners to relate new information to exist- ing knowledge structures.
"Compare and contrast the upcoming information with what you have learned in sim- ilar topics. Look for differences, similarities, and connections."
A.2 ALGORITHM
Algorithm Contextual Fine-tuning (CFT)
Require: pretrained model Pg. domain-speciff corpus
Draw
train set of contextual prompts C, batch
size B
1: for each training step do
2: Sample a batch texts (x)
)[= from Dra
3: for each text (0()) in the batch do
4: Sample contextual prompt c(0) uniformly random from C
5: Construct the input sequence x/(() (cl0) x())
6: Set the target sequence y()) x(0)
7: end for
8:
Compute the loss:
B (0
(())
log Po
(2)
|c()) (6)
,000
B k=1
9: Update model parameters using gradients (o
10:
end for
A.3 TRAINING DETAILS
In experimental setup, use several configurations of the Llama-2 models to evaluate the effectiveness of contextual fine-tuning compared to standard unsupervised fine-tuning, Specifically, employ the Llama-2 Base model with billion parameters and Llama-2 Chat models with both billion and 13 billion parameters, each with a sequence length of 4096. All models are both contextual and unsupervised fine-tuned. For contextual fine-tuning, we employ Equation 3. For the financial news dataset, where most articles are shorter than 4096 tokens, we opt not to use a packing strategy fill up all 4096 tokens. Instead, we pad any remaining space. This ensures that each semantically distinct text is associated with its own contextual prompt. If sequence with a prepended contextual prompt exceeds 4096 tokens, we simply truncate the excess, and the truncated text becomes the first text following the contextual prompt in the next example. The models are trained for one epoch with a batch size of 128 and a learning rate of 2e-5. To assess the efficiency of CFT, carefully measured the computational resources required for our experiments and compared the overhead introduced by incorporating contextual prompts. Below are the details of our computational setup and findings. We utilized the Fully Sharded Data Parallel (FSDP) training to efficiently distribute the model across multiple GPUs. Training was performed using the b (Brain Floating Point) data format. We implemented Flash Attention 2 (Dao, 2024). All training was conducted with 8 NVIDIA A100 GPUs. With the above configuration, we achieved a training speed of approximately 55, 188 tokens per second, measured using the Llama tokenizer. The fine- tuning required a total of approximately 111.11 11 GPU-hours to complete. Incorporating contextual prompts increased the total training time by approximately 0.89 GPU-hours, resulting of 112 GPU-hours. Each contextual prompt added only about 0.8% to the length of each training example on average. This slight increase in input length led to less than a 1% % increase in total training time.
total
B OPEXT
B.1 MDPI JOURNALS DETAILS
See Table for the full token breakdown.
18

---

Published as a conference paper at ICLR 2025
Journal Category
Number of Journals
Number Tokens
Allergies
25
140,865
Antibiotics
3604
26,572,807
Antibodies
380
3,248,253
Behavioral Science
1047
7,553,809
Biologics
32
261,410
Biomedicines
3909
34,783,559
Biomedical Informatics
26
201,656
Biomolecules
5861
54,189,205
Biotechnology
43
301,367
Brain Science
4205
31,937,526
Cancers
16218
444,4182662
Cardiogenetics
42
210,720
Clinical Medicine
16063
104,432,430
Clinics and Practice
499
1,395,833
Clinical and Translational Neuroscience
21
196,310
Current Oncology
759
4,458,455
Dermatopathology
78
289,595
Diabetology
45
290,812
Diagnostics
5321
33,576,475
Diseases
455
2,596,123
Endocrines
73
437,962
Environmental Research and Public Health
43763
306,603,512
Epidemiologia
60
416,064
Gastroenterology
66
311,194
Gastrointestinal Disorders
103
566,192
Healtheare
3940
24.272,007
Hearts
68
398,487
Human Life Science and Medicine
31
143,041
Immunological Research and Clinical Applica- tions
57
428, 144
Livers
30
217,950
Medicines
557
3,790,942
Medical Sciences
457
3,028,605
Oral
48
232,320
Pharmacy
977
5,240,834
Uro
38
170,764
Vaccines
3647
28,19 28,197,071
Viruses
8941
75,109.572
Table 6: Details of MDPI journals used in the dataset. The dataset comprises 121.489 biomedical journal articles covering 37 diverse topics. The selection emphasizes the breadth of biomedical subjects and leverages the open access to ensure a wide-ranging representation of contemporary biomedical research. We use a tokenizer for 5-t-3. to count the number of tokens.
19

---

Published as a conference paper at ICLR 2025
B.2 MEDICAL TEXTBOOKS DETAILS
See Table for the full token breakdown.
Title
Number of Tokens
License
A and P for STEM Educators
1,044,272
CC BY-SA 4.0
Acid-base Physiology
120,400
BY-SA 2.0
Advanced Human Nutrition
104,889
CC BY-SA 4.0
An EKG Interpretation Primer
22,766
CC BY 4.0
Anatomy and Physiology
773,297
CC BY 4.0
Anatomy and Physiology Laboratory Manual
25,555
CC BY 4.0
Atlas of Otolaryngology. Head and Neck Opera- tive Surger
792,936
CC 3.0
Biology
824,597
BY CCBY-NC-SA CCBY-NC-SA 4.0
Cell Biology, Genetics, and Biochemistry
47,612
Chemistry- Theory, Analysis, Correlation Computational Cognitive Neuroscience
176,091
118,470
CC 3.0
Concepts Biology
355,836
CC 4.0
Contemporary Health Concerns Fluid Physiology
99,534
CC CC 4.0
47,128
CCBY-NC-SA 2.0
Foundations of Epidemiology
61,369
CCBY--C
Health Case Studies
88,799
CC BY-SA 4.0
Human Anatomy
783,948
CCBY BY-NC-S 4.0
Human Anatomy for Kinesiology
273,539
Lifetime Fitness and Wellness
145,720
CC 4.0
Medical Terminology for Healthcare Professions Microbiology
224,894
CC BY 4.0
584,182
BY 4.0
Neuroscience
34,240
CC 4.0
Neuroscience for Pre-Clinie CleiCcina Students
913
CCBY-NC-SA
Nursing Fundamentals
364,315
BC
Nursing Pharmacology
213,154
BC
Nutrition: Science and Everyday Application
205,535
CC BY-NC CC BY-NC--A 4.0
Principles of Pharmacology
69,744
Remix: Women's 's Health
63,331
CC BY CC BY 4.0
Vital Sign Measurement Across the Lifespan
69,031
Table 7: Comprehensive list of the 29 open-source medical textbooks incorporated into the dataset. We deliberately selected these textbooks provide structured medical knowledge organized for educational purposes, aligning with our objective of using contextual fine-tuning to enhance the learning processes of language models. We use tokenizer for gpt-3.5-turbo count the number of tokens.
B.3 DATA PREPROCESSING
MDPI Journals The journal articles were originally XML format. We converted these docu- ments into plain text (TXT) files, focusing extracting relevant sections that contain substantive content. Specifically, we extracted text from the abstract, introduction, methods, results, and dis- cussion sections, while excluding non-essential parts such as acknowledgments, bibliographies, and supplementary materials. Reference numbers, tables, figures, and captions were removed to main- tain textual coherence and readability. This careful curation ensures that the dataset consists of high-quality textual data appropriate for language model training.
Medical Textbooks The textbooks were originally in PDF format. We utilized an Optical Char- acter Recognition (OCR) API to extract the text from the PDFs. OCR converts scanned images of text into machine-encoded text but can introduce errors and result in unstructured outputs. To address these issues, we employed ChatGPT to assist cleaning and organizing the extracted text. We provided ChatGPT with specific instructions:
20

---

Published as a conference paper at ICLR 2025
"Please edit and refine the following uncleaned and unstructured excerpt from medical textbook. Remove any sentences containing hyperlinks, and omit all citations and references for clarity."
Using ChatGPT for text cleaning offered an efficient means to process large volumes of OCR- extracted text, correcting errors and improving overall readability (See Figure for an example). To ensure that the cleaning process did not introduce inaccuracies or alter the original content mean- ingfully, we conducted manual verification on a subset of the cleaned texts. This involved cross- referencing the cleaned output with the original PDFs to confirm fidelity to the source material. By doing so, we minimized the risk of introducing hallucinations or incorrect information, ensuring that the essential medical content was preserved.
B.4 LIMITATIONS
For the texthook data, since the textbooks were originally in PDF format, we used an Optical Char- acter Recognition (OCR) API to extract the text. Despite careful processing, OCR can introduce typos or parsing errors, especially with complex formatting or specialized terminology. To mitigate these errors, employed ChatGPT to assist in correcting potential mistakes. While this approach improved the overall quality, some errors may persist. We conducted manual spot checks to identify and correct errors where possible; however, given the dataset's size, complete manual review was impractical. Regarding the MDPI journals, they have a shorter average peer-review period (approxi- mately 32 days) compared other publishers. While this expedites the dissemination of research, it may affect the depth and rigor of the review process. The shorter review time could lead to variations in article quality, with some papers potentially not meeting the highest standards of scientific rigor. Additionally, relying primarily on MDPI journals may introduce a source bias. We acknowledge that including journals from a wider range of publishers could enhance the dataset's balance and representativeness.
C EVALUATION
C.I DEBIASING
Following the notation from Robinson Wingate (2023), We approximate the debiased prediction probability for each option's content using the following:
Paetiosed(o; x) = I
Σ IEI
Poreervedddgrii I 1, x²)
(6)
Where de denotes the default-ordered option IDs (e.g., A/B/C/D), og is the corresponding option content, denotes the question, is the default imput of option IDs, and option contents. For n
number options. denotes permutation of 21, ..., n) and I denotes a of possible Is. dg.(0) denotes the corresponding option ID for äth default option content in 7-permuted setting. We then choose the option with the highest debiased probability calculated using Equation
C.2 PROMPTING
See Table for the examples of prompts used in the evaluation.
D ABLATIONS
D.I ABLATING TRAINING SCHEMES
In this section, focus ablating the training schemes across fixed model size. We provide results on Llama-2-7B Base and Instruct versions.
Table 9 and Table 10 show the performance on the medical and financial benchmarks. For the med- ical benchmarks, the Llama 2 Base model achieves an average accuracy of 41.34% With CPT, the average accuracy increases by 1.06%, reaching 42.4%. While the improvement with CFT alone is 0.94%, the CFT + IFT approach yields a significant improvement of % CFT+IFT 2.95% com- Base pared to 1.91% for CPT + IFT. Similar trends are observed in the financial benchmarks. CPT and CFT improve the baseline performance by 0.16% and 11.82%, respectively. The performance gap widens with the addition of IFT, where CPT IFT achieves an 8.67% improvement, and CFT + IFT surged by 36.28%. The advantage of CFT is also evident in the Llama 2 Chat model, which under- went both instruction fine-tuning and training with Reinforcement Learning from Human Feedback
21

---

Published as a conference paper at ICLR 2025
Table Prompt template examples for the evaluation. For the multiple-choice questions, {ANSWER) corresponds to the ground truth option ID.
Task
Template
BioMed
following muitiple chcice question about nedical knexledge. Output a singie option trom tae tour options the tinal answer. Question: [OCESTION]
Modical MMLU
(A) (OPTION X) (8) (OPTION B) (0) (OPTION c) 101 (OPTION D) The answed to the question is (ANSWER)
following
multiple choice question
nedical knowledge, options the tinal answer.
Salve it in step-by-step fashion.
MedQA
output singie option bron: tae Question: [QUESTION]
(A) (OPTION A} (B) {OPTION B) (C) {OPTION c) 101 (OPTION D}
The
the question is (ANSWCR)
Finunce
Analyze the sent of this atatement extracted from Financial news article. Statement: (STATEMENT)
RQQA SA
Sentiment: (SENTIMENT)
Classity each sentence excracted from Minancial news into either "causal' or "noise' indicates
based
relationship between finarelat Please return only the cazegory "acise' oz fesusel'. (TEXT) Anawer: (ANSWER)
Causal20
The potential ial categories are 'Finance', "Technology', "Jax Accounting BBsinesss Management', 'Induatry', , and Goverrment Controla". Your resporse should only include the category that best fits the headl ine. (TEXT)
MultiFin
Answer: (ANSWER)
Accurncy (T)
Llama 7B
Asatomy
Clincaal Knowledge
College Medivine
Gemelis
Professonal Medicine
M88OA
Average
41.34
Bise
Rise (CFT)
41.32
38.87
46.12
Bise (CPT's JPT)
29.91
45.47
42.71
37.79
49.37
41.59
35.95
43:25
Bose (CFT+ + IFT)
51.11
46.37
42,80
40.10
50,00
42.74
36,99
44.29
Chat
48.07
46.3
18.61
39,02
19.00
48.90
38.55
45.05
Char
47.17
50.50
46.32
Chut
Table 9: Comparative effectiveness Contextual Fine-Tuning (CFT) on medical benchmarks (zero- shot). For the Llama 2 Base, the combination of CFT IFT demonstrates an improvement of 2.95%, surpassing the 1.91% 9 improvement seen with CPT + IFT. the Llama Chat models, CFT alone leads to 2.76% improvement, which is notably higher than the 0.91% improvement achieved with CPT.
(RLHF). In the medical domain, CFT leads to a 2.76% improvement over the 0.91% % improvement from CPT.
These results underscore the efficacy of CFT, particularly when combined with IFT, suggesting that LLMs require a robust alignment instructional prompts and an understanding of underlying semantics.
D.2 ABLATING CONTEXTUAL PROMPTS
The core aspect of our study involves examining the impact of contextual prompts on model perfor- mance, specifically through the lens of the informational gradients the contexts provide. We conduct an ablation by introducing negative contextual prompts, which are designed to mislead the model by suggesting that the following information is incorrect. We use the following negative contextual prompts.
"Ignore everything you know about medicine. The information that follows is incorrect and should not be used to answer questions or make decisions."
"The following medical information is both true and false. Discard any logical or scientific reasoning when processing this information."
22

---

Published as a conference paper at ICLR 2025
FIQA
Causal 20
Multifin
Average
Ilama 7B
F1
F1
45.00
21.55
27.89
Base (CPT)
45.48
16.92
21.75
28.05
Base (CFT)
48.25
39.44
31.44
39.71
Base (CPT + IFT)
49.16
30.74
29.77
36,56
Base (CFT + IFT)
62.53
88.24
41.74
64.17
Chat
56.40
90,40
38.74
61.85
Chat (CPT)
62.53
90.16
38.23
63.64
Chat (CFT)
67.69
90.17
46.01
67.96
Table 10: Comparative effectiveness of Contextual Fine-Tuning (CFT) on financial benchmarks (zero-shot). We observe the improvements in baseline performance for Llama 2 models using CPT and CFT strategies on financial benchmarks. While CPT enhances baseline performance by 0. 161, CFT notably increases it by 1 1.82%. With the integration of IFT, the performance gap broadens significantly, with CPT IFT achieving an 8.67% improvement and CFT IFT results in the im- provement of 36.28%.
Accuracy (1)
Llama 2 7B
Anatomm
Clinccal Knowlolge
College Boology
Collage Medicine
Modical Ciemeties
Professional Medlieine
McliAA
Averuge
(CPT)
18,13
48,87
$2.08
14.22
$4,00
46.09
40.65
47.81
(-CPT)
41.48
47.92
50.50
46.69
38.06
45.34
Llama 13B
Anatoeny
Clinical Knowledge
College Biology
College Medicine Medical Genetics
Professional Medicine McdQA
Average
Chat (CFT)
$3.33
63.21
$7.99
56,35
62,50
57.72
44,85
56.56
Chat (-CPT)
50.00
59.62
62.15
$2.89
61.50
57.17
43,09
55.20
Table 11: Medical Benchmarks (Zero-shot). The table shows the effects of negative contextual
prompts medical benchmarks. For the 7B model, performance decline of % A CFT -2.57% noted, highlighting the adverse impact of negative prompts. Conversely, the larger 13B model exhibits more moderate decline of 11.66
-CFT
"Instead of learning from the upcoming medical data, focus on memorizing the patterns of the letters and ignore their meanings."
4. "Forget all prior medical knowledge you have learned. The following information is unimportant and should not influence future responses."
5. "Do not learn make any inferences from the following medical corpus. Treat it meaningless and irrelevant to any future tasks."
Table 11 presents the results from the medical benchmarks. For the 7B model, we observe de- crease in performance with a % CFT = -2.7%, indicating detrimental effect of negative prompts. Interestingly, the 13B model shows lesser decrease of only 1.36%. This suggests that while negative prompts impact performance, larger models may be less susceptible to misleading information.
In the financial domain, shown in Table 3, the impact of negative prompts is more pronounced. The 7B model experienced performance drop of %75CFF 33.41%, and the 13B model sees decrease of 2..9%%. Despite these declines, all models undergoing negative contextual fine-tuning still perform better than those subjected CPT.
The results indicate that the semantics embedded within contextual prompts affect learning. How- ever, contrary to our initial hypothesis that larger models would more sensitive due to their enhanced semantic understanding capabilities, the 13B model exhibits less sensitivity negative prompts. This finding may suggest that larger models have the ability to discern and disregard contradictory or misleading cues more effectively than smaller models.
D.3 TEXT-ADAPTIVE CONTEXTUAL PROMPTS
Generating contextual prompts.
We generate contextual prompts automatically by instructing GPT-4o-mini to create prompts based the content of each training batch. Specifically, the following instruction template:
23

---

Published as a conference paper at ICLR 2025
"Your task is to create contextual prompt that guides the LLM's learning process during fine- tuning.
{{ INSTRUCTION }}
{{ TEXT }}"
In this template, {{ INSTRUCTION }} is replaced with sampled instruction from the following five different instructions to generate a variety of prompts:
1. "Given the following text, generate a contextual prompt that encourages a reader to focus on the main ideas and themes presented. The contextual prompt should be concise and help the reader engage deeply with the content."
2. "Analyze the text below and create contextual prompt that guides reader to think critically about the content, questioning assumplions and evaluating arguments. The prompt should en- courage the reader to consider different perspectives presented in the lext."
"Examine the text and generate contextual prompt that encourages the reader reflect on how the information connects to their existing knowledge or experiences. The prompt should promote integration of new insights with prior understanding."
"Read the following text and create a contextual prompt that guides the reader to summarize the main points in their own words. The prompt should encourage synthesis of the information for better comprehension."
5. "Given the text below, develop a contextual prompt that leads the reader to compare and contrast the concepts presented with related topies or prior knowledge. The prompt should help identify similarities and differences."
Similarly, TEXT }} is replaced with the text from each batch.
Below examples of the generated contextual prompts adapted OpenMedText:
"Critically evaluate the methodologies and findings presented in this study on PCR techniques and LeHV-5 detection. Whut assumptions underpin the experimentul designs, and are there al- ternative approaches or perspectives that could challenge or complement the arguments made? Consider the implications of these methodologies for broader scientific research and diagnostics in veterinary medicine."
"Reflect on the complex relationship between potassium channels and chemo resislance in can- cer treatment. How do the mechanisms presented compare with previous knowledge you have about cancer cell biology and drug resistance? Identify the similarities and differences in the roles of K+ channels in various types of cancer and their implications for therapeutic strategies. Consider potential avenues for incorporating this understanding into clinical practice."
3. "Consider the findings school breakfast participation and the impact student health from multiple perspectives. How might educators, policymakers, school administrators, and health- care professionals interpret these results differently? Reflect how each stakeholder could use this information to improve student health and educational outcomes in their respective roles."
As shown in Table 5, the model trained with the automatically generated, text-adaptive contextual denoted as TextAAappCFT- achieves an average accuracy 46.31 %, outper- forming both the baseline Chat model (45.05%) and the Chat (CPT) model (45. 96%). Although it does not surpass the Chat (CFT) model with manually designed prompts (47.8 .17.81 %e,, these find- ings indicate that text- text- adaptive prompts can effectively enhance model performance across medical benchmarks. This suggests that the original manually crafted contextual prompts are not unique solution, prompts that provide meaningful semantic content can guide the model's learning by influ- encing gradient updates during training.
Results.
E ADDITIONAL EXPERIMENTS
E. SCALING UP CONTEXTUAL FINE-TUNING GEMINI- GEMINI-L.5-FLASH
24

---

Published as a conference paper at ICLR 2025
To further validate the effectiveness of contex- tual fine-tuning (CFT) on larger. more capa- ble models, conducted additional experi- ments using Gemini-1.5-Flass
ROQ
Causal 20
Multiin
Llama
Average
Chat (CFT)
67.69
90.17
46.01 43.96
67.%6
Chat (-CFT)
59.53
90. 16
64.55
These exper- iments served test whether the benefits of CFT observed on Llama models extend if the efficacy of CFT scales with model capability.
Llama 13B
FIQA
F1
Causal 20
Multina
Avernge
Chat (CFT)
70.55
89.87
50.94
70.45
Chat (-CFT)
Figure 3: Financial Benchmarks (Zero-shot), This ta- ble presents the impact of negative contextual prompts financial benchmarks. It shows notable perfor- mance drop for the 7B model, with decrease of %AcFT CCT -3.11% and smaller yet significant reduction for the 13B model at - 2.39%..
Experimental setup. Due to API limitations that restrict fine-tuning data to maximum of 500 examples with 5000 characters per exam- ple, we developed targeted approach max- imize the utility of this constraint. Rather than randomly sampling training examples, first identified the most challenging questions from the MedQA dataset by evaluating the base Gemini-1.5-Flaar model and isolating questions an- swered incorrectly. This filtering process yielded 458 questions for our fine-tuning dataset.
For each question, we generated corresponding educational content using a structured approach. Specifically, we prompted Claude 3.7 Sonnet (Anthropic, 2025) with the following template:
"I have the following medical question from MedQA USMLE exam to prepare for:
Q: [QUESTION]
A: (OPTION A} B: (OPTION B} C: {OPTION C) D: (OPTION D) Answer: [CORRECT OPTION). (ANSWER)
Please provide: A thorough textbook-style explanation written in clear, connected paragraphs that build upon each other. Include relevant clinical correlations and physiological mechanisms through- out the text. Present this as cohesive educational yet concise resource similar to what might find in high-quality medical textbook structure. The output should have less than 3000 characters."
This approach generated domain-specift educational content for each question, creating corpus suitable for fine-tuning experiments.
Text-adaptive contextual prompts. Following the principles established in Section D.3, we gen- erated text-adaptive contextual prompts tailored to each educational content piece. The prompts were created using a template that leverages one of the learning theories discussed in Section A.1:
"Based on the following question-ansse answer pair and its related educational content: QUESTION: [QUESTION]
A: (OPTION A} B: {OPTION B} C: (OPTION с} D: (OPTION D}
Answer: [CORRECT OPTION). (ANSWER)
EDUCATIONAL CONTENT: (EDUCATIONAL CONTENT)
Generate very concise contextual prompt that will enhance learning effectiveness. The prompt should:
1. Follow the style of [select one learning theory approach: Application of Knowledge/In-Dee Depth Exxploration/Refletti Thinking/Creative Infepreaaionn/Smmnariiiiooo and Synthesis/Focus on Key Concepts//ontexttall Understanding/Critiall Analysis/Question Based Learnin/Cooppara tive Learning]
Identify:
The fundamental concepts that must be understood
Critical facts that require focus for mastery How these elements connect to clinical reasoning application
Be formatted as directive that encourages active engagement with the material (approximately
1-2 sentences)
Frame the learning in a way that facilitates long-term retention
The contextual prompt should help the learner not just memorize information but develop deeper, more applicable understanding of the medical concept to correctly answer the question using the educational content. The output should only contain the concise contextual prompt with 1-2 sen- tences."
25

---

Published as a conference paper at ICLR 2025
Accuracy (1)
Llama-2 13B Chat
IFEval
MMLU
MMLU-Pro
Base
46.7
47.8
18.7
CPT
45.9
48.3
16.5
CFT
45.7
47.9
16.4
Table 13: Performance Comparison of CFT, CPT, and Baseline Models on General and Instruction- Following Benchmarks (MMLU, MMLU-Pro, IFEval). The results demonstrate that contextual fine-tuning (CFT) maintains general knowledge with minimal degradation.
Then, using the losses defined for CPT and CFT (Equation 3), we fine-tuned Semini-1.5-Flas for 50 epochs.
Results. Table 12 presents the performance comparison hetween the two fine-tuning strategies on the filtered MedQA questions.
The results demonstrate that CFT outperforms CPT by 6.71%. This substantial improvement suggests that as model scale increases, the effi- cacy of CFT is amplified. These results provide further evidence that larger models, with their enhanced reasoning and follooii capabilities, are better equipped to leverage the semantic signals from contextual prompts.
Model (Method)
Accuracy (%)
Gemini-IL5-Flash (CPT) Gemini-1.55
37.18
(CFT)
43.89
Table 12: Performance comparison contextual fine-
tuning (CFT) and continued pre-training (CPT) on 1.5-nah evaluated on filtered MedQA ques- tions.
E.2 BASELINE COMPARISON
We compare CFT with AdaptLLM (Cheng al., 2024), a domain-specific continued pretraining method that enhances knowledge by converting corpora into reading comprehension format for fine-tuning. As shown in Table CFT consistently outperforms AdaptLLM across all tasks in the biomedical benchmarks. In contrast, AdaptI.LM generally underperforms on our dataset. We attribute this discrepancy to differences lizes PubMed abstracts to create reading comprehension tasks. Abstracts typically provide concise summaries of articles, making it easier to generate meaningful question-answer pairs. In contrast, our dataset comprises full-text articles from MDPI journals and textbooks, where not every para- graph is suitable for question-answer generation. This limitation may reduce the effectiveness of AdaptLLM's approach our data. This suggests that method is more efficient, simpler, and data-struucture-annosie compared to AdaptLLMM.
the datasets used. original AdaptI..M paper uti-
E.3 EVALUATION ON GENERAL AND INSTRUCTION-FOLLOWINS BENCIIMARKS
We evaluate the OpenMedText fine-tuned Llama-2-13B model on general benchmarks and instruction-following benchmarks: (1) Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2021), (2) MMLU-Pro (Wang et al., 2024), and (3) Instruction-Follloiing Eval (IFEval) (Zhou al., 2023). As shown in Table 13, fine-tuning does not cause catastrophic for- getting; the model's general knowledge is only slightly diminished, and the capabilities of the base model largely retained. While CPT is marginally more robust knowledge degradation than CFT, the performance difference is minimal and CFT demonstrates stronger in-domain performance, as evidenced by Table 1.
F UNDERSTANDING CONTEXTUAL FINE-TUNING WITH SYNTHETIC DATA
Setup. Briefly, Garg et al. (2022) aim train transformer model capable of in-context learning of a function class F. Their goal is to demonstrate that after sufficient training, transformers can approximate any function f E by conditioning on a sequence of in-context examples inference time. Building upon their framework, we adopt their pretraining setup to train model that learns a class of function F. However, our objective differs that we investigate how contextual fine- tuning affects the pretrained model's ability to learn new function class G. Specifically, consider a function class F. our initial goal is to train a model that can learn functions f E F such that, for most functions, the model approximate f (Iquery) ) for query input Iquery by conditioning a prompt sequence containing in- in-context examples. Formally, let Dxx be a distribution over inputs, and let D, be distribution over functions in F.
26

---

Published as a conference paper at ICLR 2025
Now, consider learning new class of functions G, where each g E is composition of f with another function h from a distribution Du, that is: 9 = {9 9 g(x) = h(f(z)), E Dx)- We can draw an analogy between this setup and the fine-tuning of LLMs in specific domains. In this analogy, text can be viewed as samples from some distribution Dx over inputs, and the function class FF represents the L.M'' ability process and understand these texts. Learning new function class G corresponds to adapting the model to perform specific tasks. In the biomedical domain, this might be extracting diseases from electronic health records or answering medical questions. If the model already has the capability to compute fx) (i.e., process and understand the text), we hypothesize this can aid in learning the composed function g(x) = h(f())).
Pretraining. We first train model to learn the function class F with respect to the distributions Dj over functions and Dx over inputs. We construct random training prompts P which is a se- quence P = (I1, f (211;.... ...)))....,... Fg, [zx)), where the inputs I;8 drawn independently from Dx, and f is drawn from Dr. We then train a model to predict every f(z;) based on a set preceding in-context examples. Specifically, let p denote the prompt prefix containing i in-context examples and the (i + 1)-th input p ( f(1)), ) I2, (x22)...... Li, (x;), Ti+1) 1), we train transformer model Mo by minimizing the expected loss over all the prompt prefixes:
min, Ep
e (Mo(()))
k
where l is the mean squared error loss. In our experiment, F is the class of linear functions, that is, F {f f(z) = w x, w E RR), where the weight vectors w are sampled from A (0, La). We let Dx be the isotropic Gaussian distribution L2) Garg et al. (2022) show that after sufficient training. a transformer model can predict (I almost perfectly when there are more than 20 in-context examples.
We now extend their setup to fine- suee-tnning the pretrained transformer to learn a novel function class G. We consider two types of functions h(-) to construct G:
Fine-tuning.
1. Polynomial combination: G {g g(x) = f(x) + fz))
2. Multiple linear relationships: 9 = (g g(x)) fxx + w w2 E R2).
We fine-tune the pretrained transformer on these different function classes separately, using differ- ent training strategies: Contextual Fine-Tuning (CFT), Continued Pretraining (CPT), and Negative Contextual Fine-Tuning (NEG-CFT) which is an ablation of CFT with negative contextual prompts intended to provide non-helpful potentially misleading information. We now describe how construct the input prompts for the different fine-tuning strategies:
CPT: We fine-tune the model prompts that contain only the inputs n; and their outputs com- puted using the composed function g(z), specifically:
Pert (x1, ((1)) 22, 9(22) (FFD(
CFT: We provide the model with additional contextual information by including the original function outputs f f(x) in the prompt. The prompt structure is then:
PeFt = (I1, f(z1), ø2, 22, fI2)) .......,, zk. (**), x1, g(x1), x2 g(.2),..., g(x)))
Here, the initial sequence (#1, (z1), ), x2, f fz2)...... ...,) Dke f(*x)) )) encodes the semantic informa- tion necessary for learning the transformation introduced by h, facilitated by conditioning on the contextual prompts.
To assess the impact of the contextual prompts, we introduce NEG-CFT, where we replace the original function outputs f(x,) with random values sampled uniformly from [0, 1]. The prompt becomes:
NEG-CFT:
PNEG CFT (1,, ... Fk, Pk, T1, (2))))))). g (x*)))
where rj ~ (10 1). This ablates the meaningful contextual information to evaluate its signifi- cance in learning the function class.
For each tunint strategy, we minimize the loss in Equation using the respective prompts except compute loss over g(xi+1) instead of f((+1).
27

---

Published as a conference paper at ICLR 2025
Training details. Following Garg et al. (2022), we employ a decoder-only Transformer architec- ture similar to GPT-2 Small (Radford et al., 2019), consisting of 12 layers, attention heads, and an embedding dimension of 256. The transformer's 's output is scalar. We pre-train the transformer for 500k steps with a batch size of 64 to learn the linear function class F. Subsequently, we fine-tune the pretrained model using the different training strategies- CPT, and NEG-CFF-foo 40k steps, with the same batch size. The learning rate is set to Ic-4 throughout all training phases. All models are trained using the Adam optimizer (Kingma & Ba, 2015) with default parameters. The training process aims to minimize the mean squared error loss between the model's predictions and the target outputs, as defined Equation 7.
Results. Our experiments demonstrate that CFT of the pretrained transformer offers advantages over CPT and NEG-CFT. Empirically, we observe 1) faster convergence and lower loss, 2) improved performance with fewer in-context examples test time, 3) alignment of gradients with target func- tions, and 4) value provided by the tokens within the contextual prompt.
4a and 4c illustrate that transform- ers fine-tuned using CFT achieve lower loss compared to those trained with CPT and NEG-CFT suggesting that the content of the contextual prompts in both cases better guides training dynamics when learning new function class G. In Figure 4b and 4d, we assess the model's performance at test time with normalized squared error (Mo( - g(x))²/d where d 20 is the dimensionality of the input and weight vectors. The contextual fine-tuned transformer achieves lower errors even with a small number of in-context examples in both the polynomial combination and multiple linear relationships case. This demonstrates that CFT helps the model to learn the function class G more accurately than existing training strategies.
Contextual fine-tuning improves learning dynamics. Figure
70
250
NEG-CFT
CFT CPT NEG
CPT
60
200
CFT
Squared emor
50
150
40
30
100
20
50
10
20000
40000
0
20
40
Step
In-context examples
(a) Loss vs Step
(b) Squared error vs In-context examples
10
NEG-CFT
2.0
CFT
CPT
CPT
8-
CFT
Squared enrr
1.5
NEG-CFT
Loss
1.0
4
0.5
0.0
0
20000
40000
20
40
Step
In-context examples
(c) Loss vs Step
(d) Squared error vs In-context examples
Figure 4: We compare the performance Contextual Fine- ToeeTTnnig (CFT), Continued Pretraining (CPT), and Negative Contextual Fine-Tuning (NEG-CFT) in lcarning new function classes polynomiil combination (a-b) and multiple lincar relationships (c-d). (a) and (c) show that CFT achieves lower training loss and faster convergence than CPT and NEG-CFT. (b) and (d) depict the normalized squared error versus the number of in-context examples at test time, averaged over 1,280 random prompts; CFT attains lower errors even with fewer examples.
28

---

Published as a conference paper at ICLR 2025
Cleaning process
Example input:
Clitting protuins are mainly produced
proteins "dotting particiaate coscado dotting process during andetheiaal Injury.
One important cleting factor fbbthhogen Flbringgen generates Torin when acthvated the cosgulant thrembis, which to mesh that cots blood with the
plug. Normelly, enticoegulatt fibrinolytics plasme, Howeve; during encotheiial injury, damaged colls will ralease tisuee facto; another type of dotting factor that causes cascade of thrombnn procuction that will dotting
heperit,
the Scrum a torm used to desci be p assm that has been ramoved cletiing factors, Serum stil contains albumin and globulins, which are often called ser proteins
LICENSES AND ATTRIBUTIONS
CC LOCENSED
REEIIUULLY
Curetion and Revision, Authored Boundess.com. Provided Boundles.com License: CC BY-SA: SemriuutioonSSrrddiii
CC LICENSED CONTENT, SPECIFIC ATTRIBUTTION
Blood. Provided by: ViMpedia. Located at: en sn.Wiileediaarrs /wiki/ wwlki//iood' human blood. License: CC BY-SAL SerributtionShardddie
Humren Phypiokgy/BBool phystology Provided by: Walbooks. Located et: sm.aikibookks.rrg /wiki Herman P..od, Composition License: CC BY-S8: Attribution-Shhree ShereAttee
pizsma. by: Widtonnry,
sn.wiittionary.oo wwt.://///////////////aa/a CC BY:SA:: Attribution Sharodlike
Dlycorrtten. Provided Wictioary, Located smwilttionar..er//aae www.s///ot...eeeooooooootototonn License: BC Attribution Sharedikke
Example parsed output:
Clotting proteins are mainly produced in the liver. Twelve proteins known as "clotting fac- tors" participate in the cascade clotting process during endothelial injury. One important clotting factor is fibrinogen. Fibrinogen generates fibrin when activated by the coagulant thrombin, which forms a mesh that clots blood with the assistance of platelet plug. Nor- mally, anticoagulants and fibrinolytics in the plasma, such as plasmin and heparin, break up fibrin clots and inactivate thrombin. However, during endothelial injury, damaged cells will release tissue factor, another type of clotting factor that causes a cascade of thrombin production that will overpower the anticoagulants and cause clotting response. Serum is a term used to describe plasma that has been removed of its clotting factors. Serum still contains albumin and globulins, which often called ration and Revision. Authored by: Boundless.com com. Provided by: Boundless.com com. Li- cense: CC BY-SA: Attribution- ShareAlike Blood. Provided by: Wikipedia. Located at: en.
proteins
result. Cu-
human blood. License: CC BY-SA: Attribution- ShareAlike Human Physiology/Blood physiology. Provided by: Wikibooks. Located at: en. wn..wiibbokssoo Bakkioooss.or/wikkkee wat.soii//wwaaaaaaa Composition License: CC BY-SA: AttriutionnShareAAee
plasma. Provided by: Wiktionary. Located at: hrrpiioarr..rr/wii//paaaaaaaa License:
CC BY-SA: Attribution-ShareAA glycoprotein. Provided by: Wiktionary. Located at: en. ww..kttionn..og//iii//yyyooooooo
Example cleaned output:
Clotting proteins are primarily produced in the liver. Twelve proteins, known as "clotting factors, participate in the cascade clotting process during endothelial injury. One crucial clotting factor is fibrinogen. Fibrinogen generates fibrin when activated by thrombin, which forms mesh that clots blood with the assistance of platelet plug. Anticoagulants and fibrinolytics in the plasma, such as plasmin and heparin, normally break up fibrin clots and inactivate thrombin. However, during endothelial injury, damaged cells release tissue factor, another clotting factor that triggers a cascade of thrombin production, overpowering the anticoagulants and causing clotting response. Serum refers to plasma without clotting factors. Serum still contains albumin and globulins, which are often referred to as serum proteins.
Figure 5: OCR converts a scanned PDF into text. ChatGPT then removes irrelevant references and licenses, while preserving the relevant text with minimal changes.
29


Reviews:
[{"review_id": "0W5RIWBlIv", "reviewer": "Reviewer_fQDe", "summary": "The paper presents \"Contextual Fine-Tuning\" (CFT), a new approach for fine-tuning large language models (LLMs) that incorporates contextual prompts during training. These prompts, designed to mimic cognitive strategies like critical thinking and concept linking, aim to enhance the model's understanding and adaptability in domain-specific tasks, such as finance and medicine.", "questions": "See the weakness part.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper introduces \"Contextual Fine-Tuning\" (CFT) as an extension of instruction fine-tuning to improve domain-specific learning, which is well-positioned to address limitations in traditional methods.\n\nExtensive experiments demonstrate that CFT improves LLM performance on real-world datasets in domains like finance and medicine. The method yields notable improvements over continued pretraining (CPT) and instruction fine-tuning (IFT).", "weaknesses": "I wonder how fine-tuning on a specific domain impacts the language model’s general abilities. The authors could evaluate this by testing the model on general benchmarks to assess if base knowledge and instruction-following abilities are retained or diminished after domain-specific training. This would provide insight into whether contextual fine-tuning maintains the model's versatility across tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FN368F55fH", "reviewer": "Reviewer_ULmj", "summary": "This paper examines the role of prompting in effectively fine-tuning LLMs for new domains. Based on the premise that prompting can play a decisive role at inference time, this paper proposes the contextual fine-tuning approach by which an additional context prompt prefix is added before fine-tuning/continual pertaining documents. These contextual prompt prefixes are designed based off of human educational theories and the present 5 such examples in their work. In a synthetic function approximation setting, they perform some illustrative experiments demonstrating that the choice of fine-tuning prompt can significantly impact whether new functions can be added in post-pretraining. In this setting, they also introduce an ablative method, negative contextual fine-tuning in which the model is given random information in the context prefix and they show that this performs wors. They then examine a real domain adaptive setting, involving fine-tuning an LLM on medical and financial data. They demonstrate that their method performs better than existing approaches such as continual pertaining, instruction fine-tuning, and their combination. They also show that a \"negative contextual prompt\" which suggests that the provided information might be incorrect performs worse.", "questions": "1. I would like to see some additional ablations about the contents of the context prompts that go beyond the negative CFT setup. In particular, it would be nice if authors considered prompts consisting of various writing styles (but which are not correlated to educational theories or contradiction of the documents). This could help relate this work to prior research on the personas hypothesis etc. \n2. I would like if the authors could further justify how the synthetic data setup should be viewed as comparable to the real-world setup. In particular, why are the additional tokens added in the synthetic settings input dependent while the contextual prompts used in real settings are randomly sampled independently of the contents of the document/example?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Overall this paper does examine an interesting and important problem of expanding the knowledge or capabilities of a pre-trained large language model on new domains or topics. They propose a surprising source of gains: simply adding a prompt prefix to domain-relevant documents can improve the data efficiency of learning these new domains. The experiments across two domains appear to be well-designed and show some gains. As they note in their paper, the additional annotation and computational burden of this method is seemingly non-existent, and it could be flexibly combined with many existing continual pre-training corpora and techniques. The also attempt to justify the sources of their gains using a synthetic function-approximation task and demonstrate how an additional context can  result in faster learning. Overall, this analysis is somewhat interesting and also seems well thought out. The paper is in general easy to read and understand.", "weaknesses": "One major concern is the relationship between the synthetic setup and the real-world instantiation is not well-explained and seems somewhat tenuous to me. From my reading of the paper, it appears that the only major similarity is that both methods involve adding additional tokens to the \"fine-tuning\" documents (the contextual prompts in the real-world setting and the \"original function evaluations\" in the synthetic setting. However, these seem structurally different to me: in the synthetic setting, the added context tokens actually depend on the specific inputs and functions used in the example. On the other hand, in the real-world setting, the additional context tokens are sourced from the educational prompts and randomly sampled without considering he contents of a particular example. To summarize, it appears to me that contextual fine-tuning in the synthetic experiments actually provides a significant source of additional supervision (reminiscent of COT approaches) whereas this is absent in real-world instantiation of contextual FT. This makes the connection between the synthetic and real settings somewhat dubious to me and the mechanisms behind the contextual FT remain a bit mysterious. \n\nI also find that the design of the contextual FT prompts used in the real scenario is insufficiently justified. Currently, these prompts are associated with various educational theories. However, to me this appears to be insufficient justification because it is unclear -- and unlikely-- that any parallels can be drawn between the human learning process and the way that large language models use facts. Furthermore, if I understand the paper correctly, it seems that there is no actual task supervision corresponding to these contextual prompts (i.e. for the critical analysis prompt: \"Critically analyze the upcoming information. Look for underlying assumptions, evaluate\narguments, and consider different perspectives.”, there is no actual ground-truth supervision given to the model on what the underlying assumptions/arguments in the provided information are). As a result, the mechanisms behind how these contextual prompts actually improve performance are quite unclear, and as I mentioned previously the synthetic data setup is not convincing in its relation to the real data setup. \n\nMy concern is further amplified by the insufficient ablation analysis done on the contents of the \"contextual prompts\". The authors claim that the contents of the prompt are important by their \"negative context fine-tuning setup\". However, I think that the paper could be further strengthened if they expanded their analysis to non-contradicting context prompts which *are not* inspired by educational theories. This would help me assess the justification of the context prompt design and better understand the source behind the gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "03700bH8bd", "reviewer": "Reviewer_TQCm", "summary": "This paper introduces Contextual Fine-Tuning (CFT) for enhancing LLMs' learning capabilities by incorporating educational cognitive strategies during training. The key innovation is using instructional prompts designed to mimic human learning approaches, guiding the model's semantic understanding and domain-specific knowledge acquisition. The authors demonstrate CFT's effectiveness through experiments in medical and financial domains, showing improved performance compared to standard fine-tuning approaches while requiring limited training data.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A novel contextual fine-tuning framework that combines in-context learning with gradient-based learning using educational psychology-inspired prompts\n- Theoretical and empirical analysis demonstrating how contextual prompts affect model learning using synthetic experiments with simplified models\n- Creation and curation of OpenMedText dataset, combining academic medical journals and textbooks, offering diverse training material\n- Improved performance across down-stream applications, including medical and financial domains", "weaknesses": "- For the experimental settings, the main experiments focus on comparison between CFT and CPT. To demonstrate the effectiveness, shall the comparisons also include other ICL methods, also RAG-based methods?\n- While the authors provide thoughtful prompts based on educational theories in Appendix B1, it seems to be very limited exploration of prompt optimization or automated prompt generation methods, as the prompt template seems very various and task-specific.\n- It might be beneficial to include computational cost etc. for efficiency evaluation. \n- While OpenMedText is a comprehensive dataset proposed in a research paper (not dataset/benchmark papers), more information regarding statistics, potential biases, quality issues etc. in the dataset are not thoroughly discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3aW6CtqyHQ", "reviewer": "Reviewer_eqbM", "summary": "The paper introduces a new technique, \"contextual fine-tuning,\" which is designed to enhance fine-tuning of LLMs. The core idea is to prepend special text suggesting that the reader engage in deeper ways with upcoming content. The authors provide experimental evidence meant to show that the idea works in both synthetic and realistic settings.", "questions": "My main question is whether my reading of the difference between the synthetic vs. the real-world setting is correct. If the real-world prompts contain customized information about the upcoming text (contrary to my reading), then I would say Section 4 is relevant after all.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a very simple and intriguing idea for enhancing fine-tuning: prepending a prompt asking the reader to engage deeply with upcoming content.Moreover, experimental evidence from two real-world domains indicates that this technique outperforms natural baselines in several cases.\n\nAt a practical level, the experimental results are quite promising. This technique would be extremely easy to implement, so even small improvements with it would be a win. Although the experimental data indicates that the new method  (\"Contextual Fine-Tuning\" or CFT) is not always a win over the baseline over simply continued pretraining, it does seem to produce better results in many cases. Overall, I appreciated the medical and financial domain experiments, which seemed to have an appropriate level of complexity and realism to be a good test of the method.\n\nAt a theoretical level, this is certainly interesting. I find it counterintuitive that this technique works at all: one would think that the best way for the system to do next-token prediction would be simply to ignore the prepended CFT prompt, since it has no direct connection with the upcoming text. The fact that CFT produces good results is a hint that something surprising is happening under the hood, which could lead to future theoretical advances. Moreover, establishing that prepending this type of prompt is useful seems to point to many future areas of investigation for prepending other types of prompts.\n\nUPDATE: In light of the new experiments, I have raised my score.", "weaknesses": "There is one major weakness in the paper: I am not at all convinced by Section 4. However, I think the best thing is for the authors simply to cut this section, because I actually don't think it's necessary for their argument.\n\nHere's why I don't think Section 4 is relevant. In the synthetic function setting, the prepended prompts contain customized relevant information for the subsequent prediction tasks, helpfully factoring the functions that the system is learning to compute. This contrasts with the real-world CFT case, where—if I understand correctly—the prepended prompt has no customized connection at all to the upcoming text. I am happy to be corrected if I'm mistaken on this difference, but if I'm reading correctly, I think the synthetic task just doesn't shed light on the real-world task. That said, I don't think the synthetic task is particularly necessary for the argument, so I recommend simply cutting this section completely.\n\nIn addition to this major weakness, there are some aspects of exposition I'd try to improve. The first paragraph of the intro seems unnecessarily \"hype-y.\" I actually think you could cut this paragraph and the paper would be fine. Also in the abstract and intro, I would say it's not clear to me that this method is necessarily \"learning to learn\": that might be one explanation, but there may be others. In the conclusion, the two paragraphs seem to repeat each other a bit. The authors might consider cutting the first one and just keeping the second. I also think it might be interesting for the authors, at some point, to speculate a bit on the mechanisms that make the technique work, or suggest some future experiments that might elucidate these mechanisms.\n\nTypos: \"human's\" in abstract; \"differes\" and \"initiial\" around line 245.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)