Dissecting learning and forgetting in language model finetuning
Authors: Xiao Zhang
Ji Wu
Venue: ICLR 2024 poster (2024)
Keywords: language models
domain adaptation
catastrophic forgetting
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Finetuning language models on domain-specific corpus is a common approach to enhance their domain knowledge and capability. While improving performance on domain tasks, it often brings a side-effect of forgetting of the model's general abilities. In this study, we analyze the effects of finetuning on language models by dissecting its impacts on the modeling of topic, style, and factual knowledge in text. Our method uses instruction-following LLMs such as ChatGPT to auto-generate controlled-variable text examples which we use to probe the model. Our findings reveal that finetuning results in significant shifts in the language model's topic and style priors, while actual knowledge learning only contributes to a small fraction of the total probability change. Analysis shows that the adaptation of topic and style priors behave akin to learning simple features: they are learned rapidly and require little model capacity. They are also learned independently and primarily at the beginning of a text sequence. In contrast, factual knowledge is learned stably but slowly and requires significant model capacity to learn. The research offers insights and understanding into the finer dynamics of learning and forgetting in language models, and can potentially inform future research on improving domain adaptation and addressing the challenges of forgetting in continual learning of language models.

Published as a conference paper at ICLR 2024
DISSECTING LEARNING AND FORGETTING IN LAN- GUAGE MODEL FINETUNING
Xiao Zhang & Ji Wu
Department of Electronics Engineering
Tsinghua University xzhangl 90mai manils. tss E.singhua edu. cn, e wttemmml..ssinniiiee www.nngui.............. edu cn
ae@maill...mm
ABSTRACT
Finetuning language models on domain-specift corpus is a common approach to enhance their domain knowledge and capability. While improving performance on domain tasks, it often brings side-effect of forgetting of the model's general abilities. In this study, we analyze the effects of finetuning on language models by dissecting its impacts the modeling of topic, style, and factual knowledge in text. Our method uses cnaructtionfollowinng LLMs such as ChatGPT to auto- generate controlled-varioo variable text examples which we use to probe the model. Our findings reveal that finetuning results in significant shifts in the language model's s small
topic and style priors, while actual knowledge learning only contributes fraction of the total probability change. Analysis shows that the adaptation of topic and style priors behave akin learning simple features: they are learned rapidly and require little model capacity. They are also learned independently and primarily at the beginning of a text sequence. In contrast, factual knowledge is learned stably but slowly and requires significant model capacity. The findings offer insights and understanding into the finer dynamics learning and forgetting in language models, and potentially inform future research improving domain adaptation and addressing the challenges of continual language learning.
INTRODUCTION
Large language models (LLMs) pre-trained on general corpus show impressive common-sens. knowledge, reasoning ability, and zero-shot performance on variety of tasks (OpenAl 2023 Tou-
vron et al.) 2023 Chung et al., 2022). Finetuning LLMs on domain corpus further enhances their domain knowledge and ability, substantially improving performance domain tasks (Lewkowyz jet al. 2022, Chen et al. 2021 Singhal et al. 2023). However, is also observed that finetuning language models can lead to forgetting of previously learned information (Jang et al. [022]] Chen
jet al., 22200, which is often mitigated in practice by mixing general corpus with domain data in finetuning (Rozière al. 2023 Ouyang al. 2022).
To better understand the effect of finetuning on a language model (specifically, we study "domain finetuning"of general models on domain corpus), we perform dissection analysis on how lan- guage models model different factors of text. We analysis the topic (the overall theme, e.g., "lan- guage model finetuning"), the style (the structure, paper format), and the factual knowledge (detailed factual information, e.g., methods, citation, and results in this paper) as three main components of text. As language models represent probability distributions of text, this dissection allows us to observe how they assign probabilities to text of dif- ferent content during finetuning. This gives a finer and alternative perspective compared to existing analysis based mainly on downstream task performance.
and diction, e.g., academic writing in ICLR
To understand the behavior language models, a common approach is probing language models with specifically designed examples (Srivastava et al., 2022: Lin et al. 2022). We create samples of text with specific combinations content and style and use them to query the language model's likelihood. We study open LLMs such LLaMA (Touvron et al. 2023) domain finetuned with the conventional causal language modeling recipe used in pre-training. Following a recent trend of
automatic data generation with L.MMs (Honovich et al. 2023 Ho et al. 2023), we ChatGPT to

---

Published as a conference paper at ICLR 2024
systematically generate high-quality text samples, enabling controlled-variable probing of language models with minimal human effort in data curation.
Our investigation reveals that while domain finetuning enhances domain knowledge, it also induces strong topic and style biases in the language model towards the training data, making the model much less likely to generate text with other topics and styles. More interestingly, we found many characteristics that differentiate the learning dynamics of simple topic and style biases vs. factual knowledge. The following two findings summarizes the main contributions of our analysis:
Domain finetuning leads a significant change in the topic and style priors of the language model, biasing them towards the training data. Effect caused by such bias dominates the learning and forgetting observed in finetuning. The learning of factuals knowledge only contributes to small part of the change in modeling probabilities, which offers a possible explanation of the difficulty in preserving general abilities while assimi- lating knowledge in domain finetuning.
Topic and style biases are learned like simple features, while factual knowledge are learned like complex features in finetuning. Biases are learned rapidly with strength growing with the learning rate, and they require little model capacity learn. Even consid- erable dataset debiasing only partially mitigates them. The biases are also predominantly acquired at the beginning of the text sequence, independent of other biases. In contrast, factual knowledge is learned stably, relatively unaffected by token position, Icarning rate, or data mixture. The learning of factual knowledge also requires significant model capacity available for finetuning.
Our finding suggests that domain finetuning of language models has potential for improvement in the light of a better understanding of the learning dynamics. They could also help us identify the sources of catastrophic forgetting (French, 1999) in language models in order to facilitate effective B lifelong learning of general purpose LLMs. Our data and code are are made publicly available.
METHOD
2.1 ESTIMATING CONTENT AND STYLE PROBABILITIES IN A LANGUAGE MODEL
Our method involves estimating content and style probabilities under language model by querying it with specific text examples. With a generative model of text, we can roughly decompose the probability of a document into its generating factors. In this study, we assume that z is mainly determined by three factors: topic (the main topic of text), style (the writing style), and factual (the factual knowledge included in the text):
p(x) p(topic, style, factual)
p(topic)p(factual) (opic) topic, p(topic)p( factual topic) topie)
Note that the decomposition is only approximate and may not reflect the true generating process of text. The factors, their granularity, and the order of dependence analysis of the particular factor we are interested in. To simplify the analysis, we make a reasonable assumption that the factual and style are independent given the topic.
chosen for convenience of the
Suppose we want to estimate the probability of different styles under model p: consider two doc- uments FA and Fg sharing identical topic and factual content but written in styles A and B, respectively. The likelihood ratio between these documents under becomes the likelihood ratio of the two styles (conditioned the content).
p(#A) psstyle, topic)
p(xB) p(style (tpic)
httpss//waggiiieeeee////sssssssp http:////githb.b.cooo///a/aeeeee www.fffnnteeee ssect
ved
2

---

Published as a conference paper at ICLR 2024
Now that want to estimate the likelihood of style A style B, pairs (Z )= where TiA and T& only differ in style. All documents also have the same topic. The likelihood ratio can be estimated by averaging over the dataset to smooth out its possible dependency on specific documents:
dataset of document
p /topic) log
1
log p(ria)
plstyle topic)
N
p(I(B)
N
N
logp(ria) - Σ 2=1
log ((AB()
N
N
t=1
which be easily calculated for causal language models the difference between the average cross-entropy loss on the two set of examples ) and {
1NN
The likelihood ratio between various topics
be similarly estimated by changing the order of We do not get the raw probability, e.g.. p(sports), but we can use the decomposition in Eq. //pportsspppooltiiiss (ppolitics) to learn about the topic probabilities. Though language likelihood ratio, e.g., pl sports) models do not explicitly learn topic distribution like LDA topic models (Blei et al.; 2003), they could model implicit topic variable through approximate Bayesian inference (Wang
2023a).
For the factual factor, we are interested in the likelihood ratio of factual vs. counterfactuals, e.g. p("th sky is blue")/p("thh sky is red"), because such ratio represents the modeling of knowledge in the model. Calculating the ratio would require pairs of documents that use factural and counter- factuals with the same topic and style.
2.2 MANIPULATING CONTENT AND STYLE IN TEXT WITH INSTRUCTION
LLMs
Documents that differ in only one factor, e.g., style, might not be easy to find in existing corpus. We leverage the language understanding and instruction following capabilities of cestructionfinettune LLMs to rewrite existing documents, letting it identify and manipulate the content and the style of text. We found that with appropriate prompts, LLMS such ChatGPT can generate high-quality rewrites of passage, altering the style while preserving the content and vice versa. For example, we can explicitly ask ChatGPT to change the topic, the factual, or the style of a passage, while keeping other elements unchanged (Figure
Myeloid sarcoma (MS) defined as an extramedullary mass-forming lesion composed of immature myeloid cells. a rare but well-known manifestation of acute myeloid leukemia. Pediatrics testinular MS may pose possible diagnostic challenge, issue that underscored in the few testicular pediatric MS cases reported in the literature.
Financial fraud (FF) is defined as an illicit activity involving deceptive practices the financial sector. is a rare but well-known occurrence the world of finance. The detection of complex financial fraud cases can somelimes challenge. demonstrated by the limited number reported cases in the llterature.
change topic
change style
change factuals
So, there's this thing called myelold sarcoma
Femoral carcinoma (FC) defined as an intramedullary fludd-creating structure composed of inexperienced femoral cells. It is rare but well-known manifestation immediate femoral eruption. Pediatrics tibial FC may present potential diagnostic challenge, matter that is emphasized in the few tibial pediatric cases reported literature.
(MS), which when you got mass these immature myelaid cells hanging out somewhere other than where they'r re supposed to be. It's pretty rare, but happens sometimes with acute myeloid lsukemia. Now, when it comess to kids and their testicles, MS can bit tricky to diagnose. a whole of cases have been reported,
Prompt "change topic":
Please rewnite following passage by changing topic to an topic in please keep the structure the passage strictly unchanged please directly give the oulput without any comments
Figure 1: An illustration of changing the content and the style of a PubMed abstract with ChatGPT.
The results from rewriting show that ChatGPT effectively satisfies these strict rewriting require- ments. For instance, it can produce new content compatible with the original text's structure and
3

---

Published as a conference paper at ICLR 2024
style. We found that GPT-3.5 is capable enough for this task, although GPT-4 (OpenAl 2023 pro- duces more successful rewrites for harder cases, and its performance less sensitive to the prompt.
Effective rewriting consists of generating good quality text and adhering to the instruction. We use language modeling perplexity as a measure of quality and naturalness and found that the ChatGPT generated rewrites typically have low increase in perplexity which indicates their quality. Adher- ence is measured by human judging whether the rewrite successfully complies with the instruction. We found that ChatGPT has high success rate of around 95% (see Appendix A.3 for evaluation).
RESULTS
3.1 ANALYSIS SETUP
Data. We utilize two corpus in our analysis: PubMed collection of biomedical papers abstracts,
and C4 (Raffel et al., 2020), a large corpus of web text. PubMed is commonly used in finetuning language models for the biomedical domain (Yasunaga et al., 2022] Luo et al. [222]] Wu et al. 2023). We use it as a representative of domain corpus and use to finetune LLMss. We use C4 as representative of general-domain corpus and use it for evaluation.
For probing the content and style probabilities as in Section 2.1] we use ChatGPT to rewrite docu- ments from PubMed and C4 as described in Section 2.2 To make the analysis tractable, we sample two random subsets of 1000 documents from PubMed and C4, and then rewrite the documents with ChatGPT to generate documents with their topic, style, and factual changed. The generated derived datasets are listed Table [ Instructions used for generating each dataset and examples from the derived datasets are listed in Appendix A. and A.2
Dataset
Source
Topic
Factuals
Style
Original datasets
PubMed
biomedical nonbiomedical
factual
academic
C4
factual*
nonacademic*
Derived datasets
PubMed-nonbiomedical
PubMed
nonbiomedical
factual
academic
NobMed-counterfactii
PubMed
biomedical
counterfactual factual
academic
PubMed-casse
PubMed
biomedical
casual
PubMed-rap
PubMed
biomedical
factual
rap
C4-biomedical
C4
biomedical
factual*
nonacademie
c4-counterfactuul
nonbiomedical
counterfactual
nonacademic*
C4-academic
C4
nonbiomedical*
factual*
academic
Table 1: Datasets used for probing language models. Derived datasets are generated from the orig- inal datasets by rewriting with ChatGPT. Bold indicates the factor that is changed from the original dataset. * means "mostly", C4 is a general web corpus that could contain small portion of biomedical, academic, or counterfactual text.
In the following analysis, we calculate log-likelihood ratios by subtracting the negative causal lan- guage modeling loss between derived and an original dataset as Equation 3 For example, to measure the likelihood of biomedical topic vs. nonbiomedical topic, we calculate
p (biomedical)
log nonbiomedical)
- I(PubMed) - ((ubMeed-nonbiooeddiaal)
While we focus on the PubMed corpus most parts of our analysis, also apply the same protocol to two more domain corpus, Pile of Law Henderson et al.] (2022) in the legal domain and Amazon reviews[Ni et al. (2019) the customer review domain, for comparative analysis. Results of limited experiments those two domains are deferred to Appendix D
Finetuning setup. We finetune three language models, GPT-2 XL RRdford et al. 2019 LLaMA 2 7B and LLaMA 2 13B (Touvron et al. 2023 the PubMed abstracts using conventional causal
https://pubbnet ncbi mlm. gev We the annual baseline data of 2023.
4

---

Published as a conference paper at ICLR 2024
log atiommeiicalllttonsoommciillllll pifeltualllppccuuteeffacaaal
log placademic///trrp)
log p
LLaMA 78
LLalMW 2 138
GPT-2
PubMed
15
&llgg-lkkelhoo
10-
10
20
10
0
0
1k
10k
100k
1m
1k
10k
100k
1m
1k
10k
100k
1m
Training
Training size
Training
log pttimmelicallldnonnoomeeeiii
log piscademmiccpprrnnnssdddeiii
ptTactualllpcccunteeratcaaal
LLaMA 78
LLaMA 2 138
GPT-2
20
20
llog-likllhood
10
10-
0
1k
10k
100k
1m
0
1k
10k
100k
1m
1k
10% Training
100k
1m
Training
Training size
Figure 2: Change of likelihood ratios of content and style factors with the amount of training, averaged over three runs (shaded area represents max/min values). Significant bias towards the training topic and style is observed in finetuning.
language modeling loss. We finetune models on subsets different sizes, up to million abstracts. We use both full-finetuning and low-rank finetuning (Hu et al. 2022). We use AdamW optimizer [Loshchilov & Hutter 2019) with a learning rate of 3e-6 for full-finetuning LLaMA and le-4 for full-finetunnng GPT-2 XL and low-rank finetuning of LLaMA, all with 10% warm-up and linear learning rate decay. Learning rates are selected for each model using a grid search on a validation set. The batch size is set to 64. Other details of finetuning can be found in Appendix B
3.2 THE CHANGING TOPIC AND STYLE PRIORS DURING LM FINETUNING
Domain finetuning leads to significant change in topic and style probabilities. Figure A shows the change of likelihood ratios between different topics, styles and factual during finetuning. The likelihood of the dominant topic (biomedical) and style (academic) in the PubMed corpus in- creases significantly during finetuning with respect other topies and styles. This implies an increase in the prior probability of the training topic and style and a decrease of other topics and styles in the finetuned model. Comparing the likelihood ratio of styles academic/casual with academic/rap, it is clear that the probabilities of styles that are more different from the train- ing style (rap) have greater reduction than styles that are closer to the training style (casual).
log # academic style
All the likelihood ratios change monotoni- cally, with most showing an approximate log-
log [
style
log /bdiommed////nnonnimmed/) = web style
linear relationship with the amount of finetun- ing data. The topic and style prior probabilities are continually biasing towards the finetuning data. The factual/counterfactual likelihood ra- tio changes at slower rate, reflecting the learn- ing of new factual knowledge from the domain data and the forgetting of factual knowledge in the pre-training data. We show that the factual ratio correlates well with downstream question answering performance in Appendix
log placademii(//pprrat)))
biomed content
log (lacademmi(()p(rra()))
nonblomed content
log paccssemie"/linnnnnsaeeeeet
content
alloglikellhoo hood ratio
10
5
1k
10k
100k
1m
Training
Learned topic and style biases are indepen- dent. Figure shows the likelihood ratios of topics conditioned on different styles and of styles conditioned different topics. Notably,
Figure 3: Likelihood ratios of topics conditioned on different styles and vice versa (LLaMA 2 13B). Likelihood ratios of topics and styles are largely independent of each other.
5

---

Published as a conference paper at ICLR 2024
pubmed
pubmed
c4
c4
factuals
factuals
factuals
biomedical topic academic style
academic style
biomedical topic
academic sty/e
biomedical topic
academic style
biomedical tapic
10
-10-
10
0
-20
-0
-10
-10
10k
100k
1k 10k
100k
0
1k 10k
100k
1m
10k
100k
Training size
Training size
Training size
Training size
Figure 4: Ablating learning on PubMed and forgetting C4 by evaluating on derivative datasets (LLaMA 2 13B). Colored area show the loss change introduced by each factor. factuals" means switching from factual to corresponding counterfactual dataset. The rest "-" and "+" mean removing or adding a style or a topic from the dataset. The graphs show that topic and style adaptation contributes to the main part of the loss change in both learning and forgetting.
the likelihood ratios of topics are changing similarly under different styles and vice versa, except for very small training sizes. This suggests that the learned topic and style biases in finetuning generally independent of each other.
This independence would allow to drop the conditioning in the likelihood ratios in Equation 2 and let us study the change of topic and style probabilities separately.
3.3 ABLATING LEARNING AND FORGETTING IN LM FINETUNING
Evaluation on derived datasets allows ablate the effect of learning and forgetting in language model finetuning by introducing or removing one factor at a time. Here, we use learning to refer the loss reduction on the domain corpus and forgetting refer to the loss increase on general corpus (which roughly represents the data distribution in pre- training) in language model finetuning.
Figure 4 shows the ablation results, with learning measured on PubMed and forgetting measured on C4. The adaptation to biomedical topic and academic style contributes to the main part of the loss reduction on PubMed and loss increase on C4. This shows that the change of topic and style prior probabilities is the main cause of the observed learning and forgetting in language model finetuning.
However, the goal of finetuning on domain corpus is usually acquiring domain knowledge rather than adapting to the topic and style of the domain text. We can see that the learning and forgetting of factuals is steadily increasing with the amount of training data, although it only contributes to small portion of the total loss change. This shows that adaptation to domain topic and style is significant and probably unavoidable side effect of domain finetuning. This overly strong adaptation one possible reason for the catastrophic forgetting observed in the finetuning of language models.
3.4 CHARACTERISTICS OF BIAS LEARNING AND KNOWLEDGE LEARNING
We next delve into the distinct characteristics of bias learning and factual knowledge learning during language model finetuning.
Topic and style biases are most significant on the first few tokens and are learned quickly. To look at more details on how each factor affects the probability of document, we compute likelihood ratios separately for tokens at various positions within the text. Figure 5 shows that the likelihood ratios are clearly changing in different ways for the first few tokens and later tokens. The topic and style biases are much more significant at the beginning of the document (position <10) and are quickly learned with 1-10k documents. This implies that unconditional generation, the finetuned language model will be much more likely to generate text with the topic and style of the finetuning data by preferring those topics and styles early in generation.
For later tokens (position > 100), the topic and style biases are much weaker in comparison but are growing steadily with increased training data. The biases in later tokens seem quite consistent re- gardless of position till the end of text, thus contributing significant change to the whole document's probability. This part likely affects the conditional generation of language models by slightly biasing the generated text towards the topic and style of the finetuning data in each generation step.

---

Published as a conference paper at ICLR 2024
pifactual/ppccumnerfttttual
pthommdialllppnnnneommeeaaa
piacademicliptrap)
A Aog sog-lleithood ratio
2
Training sizes
1k
10a
10
10ah
10a
10
20
10a
101
10
10k
30k
og-llkaalhood rot
og p//actualllpccummeerstttua
log pttiommdicalllccnnaimmeaaaall
log pacaaaaaaiiiiiiicccccccccccccc
placademichprnnnasssdeddd
100k
1m
10
101
10
10a
10
10
10
10
10a
Position
Position
Posit
Figure 5: Likelihood ratios by tokens at different positions in the document (LLaMA 2 13B). Inside circles are zoomed-in views the curve at around position 10a. Topic and style biases are most significant at the beginning of sequence and are learned quickly with little training data.
Compared to topic and style biases, the learning of factuals appears more uniform across positions. This is because factual information can appear at any position, and there can be independent appear- ances of multiple factuals within document. The difference in learning speed is likely that topic and syle are simple features that are easier to learn, and that the model may alrady seen these features during pre-training on general corpora, unlike factual knowledge which more domain-specific.
Topic and style biases require minimal capacity to learn, knowledge learning requires much more. To examine the different natures of bias and knowledge learning, we finetune LLaMA 2 7B with variable numbers of trainable parameters simulate different capacities available during fine- tuning. In fill-finetuning, all parameters are tunable. In low-rank finetuning, only a small number of parameters are tunable, controlled by the rank r. For example, for r and 1, the number of tunable parameters is 0.3%, 0.07%, and 0.04% of the total model parameters.
Figure compares the change of likelihood ratios with full and low-rank finetuning at different r. Interestingly, topic and style biases are learned comparably to full finetuning with just 0.02% of tun- able parameters. On the other hand, factual learning significantly hindered by low-rank finetuning at large training sizes. This suggests that the changes of topic and style probabilities are simple biases that only require adjustments in a low-dimensional subspace of the model's representations, whereas factual knowledge learning may encapsulate encoding large number of complex patterns which requires much more model capacity.
(Side note: the learning of factuals cause decrease of ((PubMed) therefore is also affecting the topic ratio p(b p(biomedical)/ (onnioomedical) on PubMed. When capacity is limited, the topic ratio and factual ratio simultaneously reduce Pubmed in Figure
Topic and style biases magnify with learning rate, knowledge learning does not. We also examine the effect of learning rate on the learn-
full
r=8
r=2
r=1 (attention an y)
log pittiemedilerronniimee))
log piffecuulllplcountersattuu
plecademichpp(rap
placstemicclshnonsseeemm)))
ing of different factors. Figure Figure7[shows shows that the
PuoMed
C4
learned topic and style biases increase with the
log-lkklllaadd ratte
15
20
learning rate and are non-saturating, while the
10-
learning of factual remains consistent and does not increase with the learning rate.
10
5
This shows that large learning rate magnifies leamned bias, which is also correlated with the forgetting of general abilities (evaluated in Ap- pendix D). A smaller learning rate might suf- fice for knowledge learning and offers a better tradeoff between learning knowledge and preserving existing knowledge and abilities in domain finetuning.
0
0
1k
10k
100k
1m
1k
10k
100k
1m
ning
Training
Figure 6: Likelihood ratios under different model capacities for finetuning (LLaMA 2 7B). Larger rank corresponds more trainable parameters. Topic and style biases are learned with minimal capacity. Factual learning requires much more.
7

---

Published as a conference paper at ICLR 2024
3e-6
16-5
20-5
5%
40%
log piblomed//pinonniiomeel
og piblomeddyshoonbammee
log pitactaallycccunterratttaal
log pltactuallsccoutttttttutaaa
log placademicllltreel
log pieccdemniccpproooccceeeiii
log placademic/otraap og piocedemicipphoneceeeenii
PubMee
Bllog like rato
40
5 log--iicelinon ratio
20
20
10-
0
100k
1k
10k
100k
1m
1k
10k Training sire
100k
1m
100k
Training Size
Training size
Training
Likelihood ratios with different learn- ing rates for finetuning (LLaMA 2 7B). Topic and style biases magnifies with larger learning rate, while factual learning is unaffected.
Figure 7:
Figure Likelihood ratios of training with dif- ferent percentages of Wikipedia text (LLaMA 7B). Mixing a small portion of general text has limited effect on reducing the learned biases.
Mixing unbiased data reduces learned bias, but only to limited degree. Mixing general cor- pus with domain corpus a common strategy to avoid forgetting and over-adaptation. We examine the effect of data mixture using likelihood ratios in Figure mixing Wikipedia tex jinto the PubMed corpus. The results indicate that mixing a small portion of general text reduces the learned biases, but the reduction is limited and only increases modestly with the proportion of general text. This shows that while it is possible to reduce the learned biases without affecting knowledge learning by mixing& small portion of general corpus, eliminating considerably attenuating the biases may require a high general- toll-ttooomminn data ratio, making it very uneconomic in terms of training cost.
4 RELATED WORK
Dataset bias and shortcut learning. Datasets used in machine learning often inevitably contain biases in the data distribution (Torralba & Efros 201T). These superficial correlations in the data can be learned as a shortcut to achieve good performance on the training set (Geirhos et al.) 2020). The issue more pronounced in neural networks due to the tendency to learn simple features first, phenomenon known as spectral bias (Rahaman et al. [2019] Xu et al. 2019). By adapting to biases, language models can achieve loss reduction without learning much underlying knowledge. Such "surface learning" (Geirhos et al. 2020) is analogous to the "principle of least effort" in linguistics where language speakers generally try to minimize effort in communication (Chang 2016).
communicatie
Continual learning in language models. Finetuning pre-trained language models can improve model's performance on a new domain (Chen al.) 2021 Lewkowycz et al.] [002)] or a series of domains via continual pre-training (Gupta et al. 2023 Jin et al.) 2022 Ke et al. 2003).. Jang et al.) (2022) specifically study knowledge learning in continual pre-training. Forgetting is frequently observed in continual pre-training and all the above work implement techniques to alleviate forget- ting. Rehearsal (Chaudhry et al.) 2019), regularization (Kirkpatrick et al., (2077), parameter isolation combination of multiple methods often used. Mixing general corpus into the finetuning data also serves as a particular form of rehearsal.
(Rusu al. 2016).
Data generation with LLMs. LLMs such as GPT-3 (Brwnn et al. 2020) have been used to label examples for a variety of tasks (Liang et al. 2021 Hsieh et al. (023). The generated labels can be used train smaller specialized models as form of knowledge distillation (Hinton et al. 2015). LLMs have also been used to generate rationales and reasoning steps, enabling the transfer of rea-
soning abilities (Fu et al.) 2023 Ho al. for instruction-tumne tuning and alignment of LMs (Wang et al. 2023b; Honovich et al. (202)).
2023 Liet al. 2023). They also generates instruction data
Decomposition analysis of text. Separating the content and form has been a traditional approach
in literary theories Eagleton| (201T). Content analysis includes aspects like themes, ideas, and the narrative, while form (style) analysis deals with the use of literary devices like metaphors, tones, and the organization of the text. Mutiple linguistic theories further decompose the content of text into an overall topic and specific information, for example topic-focus articulation (Sgall et al. 1986) and theme-rheme analysis (Halliday 1994).
"20230901 dump from (ttps://dungy wimd a org English only
8

---

Published as a conference paper at ICLR 2024
In machine learning, the three components constitute invidividual topics of study. For example, topic modeling (Hofmann 1999 Blei et al. 2003) studies the topic distribution of text, style trans- fer studies manipulation of style Shen al. 2017) and how to separate style from content (Fu et al. 2018). Information extraction (Brin 1998, Banko et al. 2007) studies identifying factual informa- tion from text. Also, in document modeling, several work uses hierachical structure model the overall theme and specific information in text(Lin et al. 2015 Li et al. 2015
Nawrot et al. 2022)
in similar spirit as we did in this work.
5 DISCUSSION
Domain corpus used in language model finetuning can often exhibit significant homogeneity in topic and style, creating a statistically simple and salient feature easily learned by the model (Rahaman al. 2019). We show that such adaptation creates a strong bias in the language model towards the training distribution. The bias stably increases with the amount of training data and can overshadow the learning of knowledge. The presence of strong bias potentially makes the evaluation of knowl- edge learning more difficult, as an overly strong bias might interfere with the general reasoning abilities of the model.
Our observation shows that the topic and style biases are learned quickly and require little model capacity, which could mean that bias learning is hard to avoid in domain finetuning. Mixing general corpus in the finetuning data reduces the bias but adds significant training cost. This also poses challenge for lifelong learning of language models. For example, when LLMs are used as general purpose agents, we want them learn new knowledge from data without adapting too much to any individual data distribution.
While the current study aim uncover the learning dynamics in domain finetuning, we believe that by identifying bias learning as a major hindrance in domain finetuning and showing the distinct be- haviors of bias learning and knowledge learning, we also pointed out potential directions to improve knowledge learning and forgetting mitigation. For example, based on the observation that bias learn- ing mostly happens on the first few tokens of each sequence, we could mask out the loss from the first few tokens in the finetuning objective to expect reduced bias learning. Based on the different capacity requirement of bias and knowledge learning, in principle we could use a small low-rank adapter to learn the bias, and subtract its weights from the full finetuned model to remove the bias while keeping the learned knowledge. We leave the exploration of such methods for future work.
While our analysis leads to interesting findings on the learning dynamics of language models, it is limited in the following ways:
Limitations.
Separability of text- getteeeeerting factors: the decomposition of generating factors is only approximate and there may not be a generally agreed way to decompose. The boundary between content and style is not always clear, for example, terminology use is part of content and also part of style. Interdependence between content and style sometimes prevents changing
factor without changing the other. In most domain corpora, the separation is clear enough for our analysis, evidenced by the quality of rewritten documents.
Quality of rewriting with LLMs: several issues may limit the quality of generated rewrites. Safety alignment: requests for rewriting in the biomedical domain are sometimes rejected by LLM due safety alignment to reduce harmful outputs (Kenton et al.) 202I). Pre- training bias: LLMs may tend to generate text with certain topics or styles under a general instruction, which may create bias in the generated data. Hallucination (ji et al. 2023): LLMs have a certain probability of generating factually inaccurate content.
Data dependency: the quantitative observations would reflect certain characteristics of the corpus (for example, the style adaptation in training depends on the style distribution in the corpus). We compare with more domain corpus in Appendix [ and found our qualitative observations generalizes to other domains.
Limited training: we only finetuned a maximum of million documents (<1B total tokens). Although we observe a consistent trend of learning of different factors with the amount of training data, it may not generalize to very large training sizes.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The work is supported by National Key R&D Program of China 12212D00134022 We thank the anonymous reviewers for helpful comments and feedback.
REFERENCES
Michele Banko, Michael J. Cafarella, Stephen Soderland. Matthew Broadhead, and Oren Etzioni. Open information extraction from the web. In IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, pp. 2670- 2007. URL jiaaijjcii.. org/
Proceedings/ http/////P/ppppee// 129.pdt.
David M Blei, Andrew Y Ng. and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 993n))293-1002, 2003.
Sergey Brin. Extracting patterns and relations from the world wide web. In The World Wide Web and Databases, International Workshop WebDB '9, Selected Papers, volume 1590 of Lecture Notes in Computer Science, pp. 172-183. Springer, 1998. doi: 10. 10.10007100046566.1111 URL 1011011110111000000
11
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901, Curran Associates, Inc., 2020. URL httpss://proceeedngggsss////// www.innggnnnuuueeeee www.:/ddddnn...uuus............ www.ares......ppprrrrrr...e.e.e. files/ paper/2020/Fi.e http:///0...2////01....66666666. Pape..
Yu-Wei Chang. Influence of human behavior and the principle of least effort library and infor- mation science research. Inf. Process. Manag., 52(4):655-669, 2016. doi: 10.1001/j.ipm..211 12.011. URL
httgg//11111116111 1016/1 ipm. 2013.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 10486, 2019.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL https: wttpvv/aaxivvv/omm org/
03374
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pp. 7870-7881. Association for Computational Linguistics, 2020. doi: 10. 10.18633/v/2020. emnlp- emnlp-main.6 634. URL http:::/ org. 1 186538663///0 11w53//1/62222222 emnlp- caappp-maii.em 634
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu. Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery. Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff
10

---

Published as a conference paper at ICLR 2024
Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction- finetumed language models. CoRR, abs/2200.111111. 2022. doi: 10.48550//rxxi.22100.144 URL
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the AI2 reasoning challenge. CoRR, abs/1 15s30550505 2018. fhttp://arzi org/abs.....oooooo.o.ee..eeee.ee 1803. 05457
Terry Eagleton. Literary theory: An introduction. John Wiley & Sons, 2011.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences,
3(4):128-135, 1999.
Yao Fu, Hao Peng. Litu Ou, Ashish Sabharwal, and Tushar Khot. Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning, ICML 2023, volume 202 of Proceedings of Machine Learning Research, pp. 10421-10430. PMLR, 2023. URL URRLtttps//rrroeddiiim.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Ex- ploration and evaluation. In Proceedings of the Thirty-Second AAAI Conference on Artificial In- telligence, (AAAI-18), pp. 663-670. AAAI Press, 2018. doi: N0.1609/AAALV3321...1
URL
https://do/ org /10 10096699aa2.. #3211.1132 11330
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot lan- guage model evaluation, September 2021. URL https: / doi 5371628
wwa.aoorooooo.oo. worg/oo..0000 5281 /enooo..
Robert Geirhos, Jörn- Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nat. Mach. Intell., 2(11):665 673, 2020. doi: (008//42256-0200005577 URL httpss/ts:///tt orrg/
10.1038/s4220
0020000257-2
Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pre-training of large language models: How to re-warm your model? In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023. / http:::wweeeetoooeeoeeo wtt.://wwwe...eeeeo.oommmoooeoor
M.A.K. Halliday. An Introduction to Functional Grammar. Amoll, London, 2 edition, 1994.
Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Man- ning, Dan Jurafsky, and Daniel E. Ho. Pile of law: Learning responsible data filtering from the law and 256gb open-source legal dataset. In NewrIPS, URI.
2022.
les/ patep/22p///2 cc/ Datasets and
Benchmarks. html
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021. URL https: wwwprr/viwww.ee//oonnnnoooeo
3Gm
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. NIPS Deep Learning and Representation Learning Workshop. 2015.
Namgyu Ho, Laura Schmid, and Se- Young Yun. Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol-
ume 1: Long Papers), ACL 2023, pp. 14852-14882. Association for Computational Linguis- tics, 2023. doi: 18.18653/v 001118653/11/222 1/2023.acl--ooogggggm.o...oaa 830. URL https: //oo tttp:1..163...../1 2023 acl long.o..... 830

---

Published as a conference paper at ICLR 2024
Thomas Hofmann. Probabilistic latent semantic analysis. In UAI '99: Proceedings of the Fif- teenth Conference on Uncertainty in Artificial Intelligence, pp. 289-296. Morgan Kaufmann, jap ?mmnu-
1999. URL https://dslpitt wttp://aa/diisssaaaaaaeeaeeeeeee Lasm==/bartiiccee__111
ig 1d=1
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61 Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 14409-14428. Association for Computational Linguis- tics, 2023, doi: 10. 10.18653/v1/20233 awl.o:2..alloo....11 URL htt.:ttee///o...r///10116653333/ 2023 acl- long.long.... 806.
Cheng-Yu Yu Hsieh, Chun-Liang Li, Chih-Kuan Ych, Hootan Nakhost, Yasuhisa Fujäi, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Associa- Association for Computational
tion for Computational Linguistics: ACL 2023, pp. Linguisties, 2023. doi: 10.18653/w1/2023. findings-acl.507. URL httph:tt////t org//0. 18653/v1 /20233. f - ac1.507
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022. OpenReview.net, 2022. URL
pettps:/eopenreieer net /
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stan- ley Jungkyu Choi, and Minjoon Sco. Towards continual knowledge learning of language mod- cls. In The Tenth International Conference on Learning Representations, ICLR 2022. OpenRe- view.net, 2022.
net wwt.:eett/foor met:/oorum2ld=vffR53555
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv. 55(12)244: 244:12283833
2023. doi: 10.1 10.1145/3571730. URL https:://oi. org/1C.
1145/3571730
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang- Wen Li, Xiaokai Wei, Andrew O. Amold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging cor- pora. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, pp. 4764-4780. Association for Computational Linguistics. 2022. doi: 10.18653//1/2022..maam
www.2222.naal-mmii... URL
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-
training of language models. In The Eleventh International Conference on Learning Representa- tions, ICLR 2023. OpenReview.net, 2023. URL htt.sttt::/// /ttp::/:///perreieew eww....www........ wwt. het/ppffidse
GDI Ital3o
Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. Alignment of language agents. CoRR, abs/2103.1 14659, 2021. URL URL https::/arxxxi.
org/ /bg//ab//20 22t//2113... 14659
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Grabskk-Barwinska. Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):351-33526 2017.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman- Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solv- ing quantitative reasoning problems with language models. In NewrIPS, 2022. http://papers.nn nips. cttppscce/ppppe cc/paper http//appee_.leeepapp////222 ottp::/eeeeeeeaataaaaaaat--rrrrr
URL
12

---

Published as a conference paper at ICLR 2024
Jiwei Li, Minh- Thang Luong, and Dan Jurafsky. A hierarchical neural autoencoder for paragraphs and documents. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACI. 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pp. 1106-11111 The Association for Computer Linguisties, 2015. doi: 10.3115/V 107.
I Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai- Kei Chang. and Yejin Choi. Symbolic chain-of-thought thought distillation: Small models can also "think" step-by-step. In Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, pp. 2665-2679, Association for Computational Linguistics, 2023. doi: 10. 18653vVvl 2023.acl- 150. https:::///// httpttts:///o.
1ee.1....611///1 www.222........oo.. 1500
Kevin Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and
Lawrence Carin. Mixkd: Towards efficient distillation of large-scale language models. In 9th International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021. URL
net.
Rui Lin, Shujie Liu, Mayun Yang, Mu Li, Ming Zhou, and Sheng Li. Hierarchical recurrent neural network for document modeling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pp. 899-907. The Association for Computational Linguistics, 2015. doi: 10. 10.18653/11/D1-- 1106. URL /ttt::/://ool.oorg00111166///// 615-1106
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), ACL 2022, pp. 3214-3252, Association for Computational Lin- guistics, 2022. doi: 10. 00.18653/1122022..co hcll 229. URL sttpps://oi.... http://000...0110.111185.11 01/2022.
229
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, 2019. URL htttps://ppereviiee
net net/Eorum: (i..ruumiid-3KK6RRCCGG
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Yin Liu. BioGPT: generative pre- pre-trained transformer for biomedical text generation and mining. Brief- ings in Bioinformatiic, 23(6), 09 2022. ISSN 1477-4054. doi: 10.1099/bib/bbacc09. URL bbac409.
Piotr Nawrot, Szymon Tworkowski, Michal Tyrolski, Lukasz Kaiser, Yuhuai Wu, Christian Szegedy, and Henryk Michalewski. Hierarchical transformers are more efficient language models. In Find- ings of the Association for Computational Linguisticx: NAACL 2022, pp. 1559-1571. Association for Computational Linguistics, 2022. doi: 10. 122.5///
URL
11w1118633322 //0222///22222//////// :ttps://dos 1tt.:o..rr/11..1..3//11
117
Jianmo Ni, Jiacheng Li, and Julian McAuley, Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, pp. 188-197. Association for Computational Linguistics, 2019. doi: 10.18653/V1/D19. 1018. URL httpss/:::/// //dsis:://d.... oro/orrggl/..o http///....63//1/...1..
OpenAl. Gpt-4 technical report. Technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. In NeurIPS, 2022. URL. www.lttt::/// http:://////ppesr... n cc/pape f / 3lefde53bee364a otteee/eee4444110000002222aaaarn
Alec Radford, Jeff Wu. Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
13

---

Published as a conference paper at ICLR 2024
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:77, 2020. URL http:// /ttp: /m/. papers/v21/20-00 07.mmtmll
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research. pp. 5301-5310. PMLR, 09-15 Jun 2019. URL https:://pcst latps::/ppoceeddng .mlr.
press/ v9 / www.aaaaaanannn...aa 9a html
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi. Jingyu Liu. Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. CoRR, 2ttp//.........5.. 2023. doi: 10448550/arXi.2238..222
URL
2308 12950
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR,
2016. URL/ttt:://axxlv.oor www.oo/gg/////o///e/////oeeeeeee abs/1600. abs/1606.04671 04671
Petr Sgall, Eva Hajicová, and Jarmila Panevová. The meaning of the sentence in its semantic and pragmatic aspects. Springer Science & Business Media, 1986.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. Style transfer from non-parallel text by cross-alignment.. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. neurips hcc/pappr//211///0
0830-6841, 2017. URL httph::///popeeeeeeeee 2d2 1el11111261aaaa (11.1/....11111322012-------1111 -0w.:7444302277-----aaa-. html
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang. Mohamed Amin, Sami Lachgar, Philip Andrew Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Agüera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle K. Barral, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan. Towards expert-level medical question answering with large language models. CoRR, 0bs/23055096117 2023, doi: 1044850//rXii:2335.0617. URL httpts://tooi....// http:///bb...ogg................ 2305, 09617
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlüüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CoRR, aass/2206.044115 2022. doi: 0..48550/rX...2226..00 org/10. 4B5/100.4550/aaa/ arxiv 2206 04615
URLhttps: htppp://dor //doi.
Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. The 24th IEEE Conference Computer Vision and Pattern Recognition, CVPR 2011, pp. 1521-1528. IEBE Computer Soci- cty, 2011. doi: 1wL.10//CVR..2011159999 201 5995347
URL htthtts::://ddm http::/::///.ii.g/1.1111.01
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy
14

---

Published as a conference paper at ICLR 2024
Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molyhog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang. Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, 0bs/2307.099288 2023. doi: 10448550/arXx..2207.22222 09288
URL
https://doi wtt.ddii...g.....1 www.10..10000......... 485500.85500aai1
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large lan- guage models implicitly topic models: Explaining and finding good demonstrations for in- context learning. In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023a.
URL htttttsss//pppeeeeeeeeee wwwpenreieee.eee/oooeeeeeeee www.:/rrmmttoteroeoo.. wdw.hmmmd-HHcccccccck
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with ceff-generated instruc- tions. In Proceedings the 61st Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), ACI 2023, pp. 15484-13508 Association for Computational Linguis- tics, 2023b. doi: 10. 18653//3 1ttpl////222.... acl.oon.al-ooo..7.. 754.
URL. https: org/ 100 18653/
v1 v1/2022 200.2aacl222222 long. 754
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 - Demos, pp. 38-45. Association for Computational Linguistics, 2020. doi: 10.18653/v /ttp//220000mm/emeeeeeeee //doi.org//l 1 18653/v1/620/
URL https:
emn memnlp-demos.. 6
Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Yanfeng Wang. and Weidi Xie. Pme-Ilama:
Towards building open-source language models for medicine, 2023.
Zhi-Qin John Xu, Yaoyu Zhang, and Yanyang Xiao. Training behavior of deep neural net- work in frequency domain. In Neural Information Processing 26th International Confer- ence, ICONIP 2019, volume 11953 of Lecture Notes in Computer Science, pp. 264-274, Springer, 2019. doi:
22. URL
978-3-030-36708- 22
Michihiro Yasunaga Jure Leskovec, and Percy Liang. Linkbert: Pretraining language mod- cls with document links. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, pp. 8003-8016. Association for Computational Linguistics, 2022. doi: 10. 1/22
URL https:
//doi. ww..or///// 10. 18653/v1/ 2021122222..2 aalllllogg.... 551
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can
machine really finish your sentence? In Proceedings of the 57th Conference of the Associa- tion for Computational Linguisties, ACL 2019, Volume 1: Long Papers, pp. 4791 4800. As-
sociation for Computational Linguistics. 2019. doi: 10. 12118633/1//11-14422 7/doi. 1ttp://1...1/3//111//33333
URL https:
15

---

Published as a conference paper at ICLR 2024
A DATA
A.I DATASET GENERATION
Instructions used for each generating each derived datasets are listed below. A non-cherry picked example is also provided.
We found that ChatGPT (GPT-3.5-turbas is capable enough for this task with some tuning of the prompts. We also found that GPT-4 require significantly less tuning of the prompts to produce successful rewrites, and can produce successful rewrites for examples which ChatGPT fails. We use ChatGPT (GPT-3.5-turbo) with the following prompts for generating the derived datasets for analysis.
We use nucleus sampling with p=0.9 to reduce the likelihood of generating low-quality rewrites.
PubMed-nonbiomediie
Prompt:
Please rewrite the following passage by chunging its topic to an irrelevant topic in arl, finunce, education or software.
please keep the structure of the passage strictly unchanged
please directly give the output without any comments
cuateet-ounterfacttull
Prompt:
Please change the biomedical terms in the following passage into other random biomedical terms so that the biomedical knowledge disrupted. please keep the main topic and the words that are not biomedical terms unchanged please directly give the output without any comments
PubMed-casual
Prompt:
Please rewrite the following passage using casual style.
please keep the content (including all terminology) strictly unchanged pleasse directly give the output without any comments
PubMed-rap
Prompt:
Please rewrite the following passage using the style of rap.
please keep the content (including all terminology) strictly unchanged please directly give the output without any comments
C4-biomedical
Prompt:
Please rewrite the following passage by replacing its main topic with biology medicine related topic (for instance, some disease, diagnosis, drug. or treatment). please keep the style and the structure of the passage unchanged please directly give the output without any comments
C4-counterfactual
Prompt:
Please swap all the nouns in the following passage into random related nouns that every piece of information given become random and completely different from the original.
every piece of information must be changed please directly give the output without any comments
C4-academic
16

---

Published as a conference paper at ICLR 2024
Prompt:
Please rewrite the following passage using the style of an abstract of a research paper (without title).
pleasse keep all the content strictly unchanged pplease directly give the output without any comments
We randomly sampled 1000 documents from PubMed and C4 (the validation split) respectively (each document have at least 500 characters), and generated derived datasets from the samples. The generated datasets are listed in Table [
A.2 EXAMPLES OF GENERATED TEXT
PubMed (original)
According to preliminary data, seroconversion after mRNA SARS-CoV-2 vaccination might be unsatisfactory in Kidney Transplant Recipients (KTRs). However, it is unknown if seronegative patients develop at least celludar response that could offer a certain grade of protection against SARS-CoV-2. To answer this question. we prospectively studied 148 recipients of either kidney (133)
(15) grafts with assessment of IgM/lgG spike (S) antibodies and ELISpol against the nucleocapside (N) and the S protein at baseline and 2 weeks after receiving the second dose of the mRNA-1273 (Moderna) vaccine. At baseline, 31 patients (20.9%) had either fgM/lgG or ELISpot positivity and were considered to be SARS-CoV-2-PT pre-immunized, while 117 (79. I 1%) patients had no signs of either cellular humoral response and were considered SARS-CoV-2-men After vaccination, naive patients who developed either humoral or cellular response were finally 65.0%, of which 29.9% developed either IgG or IgM and 35.0% S-ELiSpot positivity. Factors associated with vaccine unresponsiveness were diabetes and treatment with antithymocytes globulins during the last year. Side effects were consistent with that of the pivotal trial and no DSAs developed after vaccination. In conclusion, mRNA- 1273 CARS-CoV-2 vaccine elicits either cellular or humoral response in almost two thirds of KTRs.
PubMeddnonbiomediic
According to preliminary data, the use of grayscale in abstract painting might be unsatisfactory in terms of conveying emotions. However, is unknown if paintings created without grayscale can still evoke certain level of emotional response. To answer this question, we conducted prospective study involving 148 artists who work primarily with either acrylic (133) or (15) paints, assessing the use of grayscale and color satu- ration in their artworks. The assessment was done at baseline and 2 weeks after the artists experimented with incorporating grayscale into their paintings. At baseline, 31 artists (20.99 (%) had already used grayscale or ex- perimented with in their artworks, and were considered pre-immmuizzed grayscale, while 117 (79.1% artists had no signs of using grayscale and were considered grayscale naive. After the experiment, prayscale-naive artists who incorporated grayscale into their paintings accounted for 65.0% of the participants, with 29.9% using grayscale as part of their color palette and 35.0% achieving grayscale effects through techniques such as cross- mrass-hatching. Factors associated with unresponsiveness to grayscale experimentation were the artists' preference for vibrant colors and their training in traditional painting techniques. Overall, the results indicate that the use of grayscale abstract paintings can elicit a response in almost two thirds of artists.
cobMeedcounteefactual
According to preliminary data, seroconversion after DNA HIV-1 vaccination might be unsatisfactory in Brain Tumor Recipients (BTRs). However, it is unknown if seronegative patients develop at least a cellular response that could offer a certain grade of protection against HIV-I, answer this question, we prospectively studied 148 recipients of either brain (133) or brain-pancreas (15) grafts with assessment of IgM//gG spike (S) anti- bodies and ELISpot against the nucleocapside (N) and the protein at baseline and 2 weeks after receiving the second dose of the DNA-7532 (GenBio) vaccine. At baseline, 31 patients (20.9% had either IgM/lgG or El.SSpot positivity and were considered to be
while 117 (79.1%) 1%) patients had no signs of either cellular or humoral response and considered HIV-I-naive. After vaccination, naive patients who developed either humoral or cellular response were finally 65.0%, of which 29.9% developed either IgG or IgM and 35.0% S-ELISpot positivity. Factors associated with vaccine unresponsiveness were diabetes and treaiment with antithymocytes globulins during the last year. Side effects were consistent with that of the pivotal trial and no DSAs developed after vaccination. In conclusion, DNA-7532 HIV-I vaccine elicity either cellular or humoral response in almost two thirds of BTRs.
PubMed-casual
17

---

Published as a conference paper at ICLR 2024
So, here's the deal: the early data suggests that people who have undergone kidney transplants (yeah, those Kidney Trunsplant Recipients, or KTRs) might not be getting a good inmune response after getting the mRNA SARS-CoV-2 vaccine. But here's the thing don't know yel: even these patients don't have antibodies, could they still have some sort of cellular response that gives them at least some level of protection against SARS-CoV-2? To find out, we did study with 148 people who got either a kidney (133 of them) kidney- pancreas (15 of them) transplant. We checked their IgM/IgG spike (S) antibodies and did ELISpot tests to see their response to the nucleocapside (N) and proteins before the vaccine and two weeks after they got the second dose of the mRNA-1273 (Moderna) vaccine. At the start, we found that 31 patients (that's 's about 20.9% of them) already had either IgM/tgG antibodies or positive ELISpot result, meaning they had some level of pre- pre-existing immunity SARS-CoV-2 The other 117 patients (about 79. 1% of them) showed no signs of either a cellular or humoral response and were considered SARS-Co SARS-CoV-2-naive After getting vaccinated, we found that 65.0% of the initially natve patients ended up developing either humoral or cellular response. Out of those, 29.9% had either IgG or IgM antibodies, and 35.0% had a positive S- S-ELiSpot result. We also looked at some factors that might be linked to not responding well the vaccine, and found that diabetes and treatment with antithymocyte globulins in the past year were associated with a lower response. The side effects were pretty much what expected based on the main trial, and we didn't 't see any development of donor-specific antibodies after vaccination. So, to wrap it up, the ARNA-1273 SARS-CoV-2 vaccine seems to trigger either cellular a humoral response in almost two-thirds of kidney transplant recipients.
PubMed-rap
Yo, check it out, here's the deal, Preliminary data reveals what's 's real,
Seroconversion after SARS-CoV-2 vax,
Might not be satisfactory in KTRs, that's 's the facts. hold up, can seronegative patients shine?
With some cellular response to draw the line, A gainst A-RS-CoV-2, can they protect?
That's 's the question we're 're here dissect. Prospective study, 148 recipients in sight, Kidney
they' 'r in the fight,
Testing IgM/lgG spike antibodies, so fly, ELiSpot against N and S, let's give try. At baseline, 31 patients were pre-imm rre-immuizeed, With lgM/lgG or ELISpot, they were baptized, But 1 117 patients showed no response at all, SARS-CoV-2 --vvee they stood tall. After vaccination, the game changed, 65.0% of the naive were rearranged, With humoral cellular response in their veins, 29.9% with IgG/IgM, the victory remains. And 35.0% with S S-ELISpot positiviry. They'r re building defenses with ferocity, Factors linked unresponsiveness, it's true, Diabetes and antithymocyte treatment can skew. Side effects aligned with the pivotal trial, No DSAs developed, that's 's a smile, conclusion, the mRNA- 1273 vaccine, Brings cellular or humoral response on the scene. KTRs, listen up, almost two-thirds rejoice, With immunity rising, you have a choice, To fight off SARS-CoVe stand tall and bold, Thanks to Moderna's vaccine, the story unfolds.
A.3 QUALITY ASSESSMENT OF GENERATED TEXT
Quality. The per-token perplexity of the original and derived datasets under LLaMA 2 13B is listed in Table 2 We can see that the perplexity of the derived datasets higher than the origi- nal datasets but not much (except for the counterfactual rewrites which produce text contradicting common knowledge, therefore can have high perplexity under language model). The generated
18

---

Published as a conference paper at ICLR 2024
text of acceptable quality and not significantly different from the original text in terms of natural-ness judged by a language model.
Dataset
Perplexity
Original datasets PubMed
5.66
C4
6.68
Derived datasets
PubMed (counterfactual)
9.29
PubMed (nonbiomedical)
9.04
PubMed (casual)
6.20
PubMed (rap)
8.13
C4 (counterfactual)
13.03
C4 (biomedical)
7.38
C4 (academic)
6.80
Table 2: Perplexity of original and derived datasets under LLaMA 2 13B.
Adherence. We also evaluate the adherence the generated text to the instruction. We randomly sampled 100 examples from each derived dataset and asked human annotators to label whether the generated text successfully comply to the instruction. "Good" means the generated text generally comply the instruction, "partial" means the generated text only comply to part of the instruction (for example, the text was successfully changed to the requested style but the content was also significantly changed), and "bad" means the generated text does not comply the instruction. The results are listed in Table 3] We can see that the generated text are generally of high adherence to the instruction. It seems that changing style has higher success rate than changing content, which could mean that identifying and changing style is easier for LLMs.
Dataset
Good
Partial
Bad
PubMed (counterfactual)
93
6
PubMed (nonbiomedical)
97
2
PubMed (casual)
100
0
0
PubMed (rap)
100
0
0
C4 (counterfactual)
97
3
0
C4 (biomedical)
94
5
C4 (academic)
96
3
1
Table 3: Adherence of generated text to the instruction.
B FINETUNING SETUP
To determine the learning rate, we perfomed grid search and train models under 100k documents with learning rates (le-6, 3e-6, le-5, 3e-5, le-4, 3e-4) for full-finetune and {le-5, 3e-5, le-4, 3e-4, le-3, 3e-3} for low-rank finetune. The final learning rate is chosen as the one that gives the lowest perplexity the validation set. We use batch size of 64.
During finetuning, we treat each document as separate sequence to keep the structure of the docu- ment for better analysis of per-position likelihoods. This is different from common LM pre-training setup where all the documents are concatenated together. All documents are truncated to a maximum of 1024 tokens.
Finetuning is performed with Huggingface's transformer library (Wolf et al. 2020), with bfloat 16 mix-precision NVIDIA A 1100 GPUs
19

---

Published as a conference paper at ICLR 2024
C RESULTS ON MORE DOMAIN DATA
To explore the effect of finetuning on other domain data, we perform similar analysis on the legal domain and the customer review domain with the setup in Section 3.1 For the legal domain, use the Pile of Law corpus (Henderson et al. 2022). We finetune LLaMA 2 7B on to 1M court opinions from the "Court Listener Opinions" subset of the Pile of Law corpus. For the customer review domain, we use the Amazon reviews dataset (Ni et al., 2019). We finetune LLaMA 2 7B on up to IM reviews of automotive products from the "automotive" subset of the Amazon reviews dataset. We also rewrite the original documents to generate derived datasets. The datasets used are listed Table 4
Dataset
Source
Topic
Factuals
Style
Original datasets Pile of Law
legal automotive
factual factual
court opinion customer review
Amazon reviews
Derived datasets
Pile of Law (nonlegal)
Pile of Law
nonlegal
factual
court opinion court opinion casual
Pile of Law (counterfactual)
Pile of Law
legal
counterfactual
Pilc of Law (casual)
Pile of Law
legal
factual
Pilc of Law (rap)
Pile of Law
legal
factual
rap
Amazon reviews (nonautomotive)
Amazon reviews
nonautomotive
factual
customer review review academic
Amazon reviews (counterfactual)
Amazon reviews
automotive
counterfactual
Amazon reviews (academic)
Amazon reviews
automotive factual
Amazon reviews (rap)
Amazon reviews
automotive
factual
rap
Table 4: Datasets used for probing language models the legal and the customer review domain.
Refer to Table for explanation of the columns.
Figure and 10 shows the change of likelihood ratios between different topics, styles and factual during finetuning the legal and the customer review domain. Similar the biomedical domain, the likelihood of the dominant topic and style in the training corpus increases significantly during finetuning with respect to other topies and styles. The Pile of Law data has a dominant topic of legal affairs and a dominant style of court opinion. The Amazon reviews data has dominant topic of automotive products but the style is generally casual and more diverse than other domain text, therefore the adaptation of style prior is less significant. This shows that the presentation of dominant topic and style the training corpus would invariably leads to strong adaptation of topic and style priors under the present language model finetuning regime.
For all the domains, the factual/counterfactuaa likelihood ratio changes at a significantly slower rate than the topic and style likelihood ratios, showing that the effeet of topic and style adaptation on text modeling probabilities are much more significant than the effect of knowledge learning.
D ADDITIONAL LM EVALUATION
Language model evaluation: general abilities We evaluate LLaMA 2 7B finetuned our anal-
ysis Hellaswag (Zellers al. 2019) ARC (Challenge Set) CCark et al. 2018) and MMLU
Hendrycks et al.) 2021). using the Language Model Evaluation Harness framework Gao et al.
2021 Zero-shot and 5-shot performance presented in Table
Hellaswag 0-shot
ARC-Challenge 5-shot
MMLU
Learning rate
Training size
5-shot
O-shot
O-shot
5-shot
Baseline
76.0
78.1
46.2
53.2
42.6
46.6
3--6
IM
76.5
77.8
46.0
52.5
42.2
46.5
le-5
1M
75.7
76.9
44.4
50.0
40.5
45.0
2e-5
1M
73.9
75.2
41.7
46.1
37.3
42.1
Table 5: Evaluation of LLaMA 2 7B finetuned PubMed.
20

---

Published as a conference paper at ICLR 2024
log pliegall/p(nonlegal)
log plautomativellllronaattooooddd
log pffectuall/pcconnteeftttaa))
log pffactual//ccounteefatttaa
log p(court opinion)/p(rap)
log p(review\/p(rapp)
log p(court opinion//p(casuul)
log pireview./p(academic)
LLaMA 7B
Arazon Review
LLaMA 7B
log llgllllllhoodd ratio
150
log likellhood ratio
40
Pile of Law
100
20
50
0
10k
100k
1m
1k
10k
100k
1m
Training size
Training size
Figure 9: Change of likelihood ratios of con- tent and style factors with the amount of training Pile of Law.
Figure 10: Change of likelihood ratios of con- tent and style factors with the amount of train- ing Amazon reviews.
Language model evaluation: medical knowledge To verify that medical knowledge is learned through finetuning the PubMed corpus, evaluate LLaMA 2 7B clinical subsets from MMLU, following the Med-PaLM paper (Singhal et al. 2023). Results (5-shot) are listed in Table 6 We further plot the factual/counterfaatual likelihood ratio and the average accuracy MMLU clinical subsets on the same graph in Figure [1] The two curves show a similar trend, indicating that the factual/counterfaatuul likelihood ratio is indeed indicator of the learning of biomedical knowledge, to the degree that MMLU clinical subsets reflects knowledge learning on PubMed.
Clinical knowledge
College biology
College medicine
Medical genetics
Professional medicine
Training size
Anatomy
Baseline
46.7
45.7
46.5
41.0
51.0
51.8
1K
45.2
46.0
46.5
42.2
52.0
51.8
10K
45.9
46.0
45.1
42.2
53.0
52.6
100K
47.4
46.4
46.5
43.4
52.0
52.9
1M
48.1
46.4
46.5
42.8
54.0
54.4
Table Evaluation of LLaMA 2 7B finetuned on PubMed.
MMLU accuracy (clinical subset) log ptTactusl//p(countetttttual)
50
A llg-llkeell ligllikllloodd ratio
10
49
Accuracy
48
47
0
1k
10k
100k
1m
Training size
Figure 11: Comparing the change the factual/counterraccuuall likelihood ratio and question answer- ing accuracy on MMLU clinical subsets.
21


Reviews:
[{"review_id": "tAmKnN5JGR", "reviewer": "Reviewer_DT5M", "summary": "- This paper presents a detailed analysis of the effects of fine-tuning of large language models on domain-specific downstream tasks/datasets.\n- In doing so, authors break down the probability distribution of a text into its fundamental factors i.e., topic, style and factual knowledge, and study the effects of fine-tuning on the probability distribution over these three factors.\n- It has been shown that in the early cycles of fine-tuning, the language model easily captures the topic and style information of the underlying text data thus introducing learning bias, which ultimately leads to an increase in the forgetting of the previous knowledge. However, the model is able to capture the factual knowledge in the later cycles of fine-tuning and also requires significant model capacity as compared to the model capacity required for capturing topic and style information.\n- Extensive experimental evaluation asserts the claims made by authors and opens a new research direction in continual learning research.", "questions": "- The C4 dataset could possibly contain the documents written in the \"academic\" style although in a different domain. Similarly, the C4 dataset could also contain documents related to the biomedical domain although having different factual information. So, it is possible that the model is adapting fast to the academic style and biomedical domain topic because it has already seen them in the pretraining data, but the diverse factual information in the PubMed dataset is new for the model, and that is why model is possibly taking time to capture that knowledge. Have authors taken this into consideration in their analysis of the finetuning process?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Quality\n\t- The motivation is well-founded and the claims are sound.\n\t- Experimental analysis is very detailed and explanatory.\n- Clarity\n\t- Paper is clearly presented and easy to follow.", "weaknesses": "- Quality\n\t- As the topic of a document can be determined by the factual knowledge it contains then it might be redundant to keep the topic as a relevant factor in the text generation process and only style and factual knowledge might suffice which then could directly align with the syntax and semantics of the underlying text respectively.\n- Significance\n\t- This paper presents a detailed technical analysis of the fine-tuning process of a language model on domain-specific downstream tasks/datasets. However, the outcomes of the study conform with the expected outcomes of fine-tuning a model on domain-specific data and hence this paper misses to provide any significant gainful insight into the fine-tuning process due to the following reasons:\n\t\t- In the PubMed dataset, as academic style is present across all abstracts with different factual knowledge, it is expected that the model will readily adapt to the academic style first before capturing the diverse type of factual knowledge.\n\t\t- Just like in topic modeling, the topic of a document is a broad sentiment and can be easily determined using a set of keywords. So, it will be easy for the model to detect/understand the topic of a document before reading the whole document and capturing the factual knowledge inside it. Therefore, it is expected for the model to easily understand the topic and style factors before capturing the factual knowledge inside it.\n\t- I am keen to hear the response of the authors on this and hope that they can change my point of view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IeQh1AMom9", "reviewer": "Reviewer_xGn4", "summary": "This paper dissects the effect of fine-tuning on learning and forgetting of language style, topic, and factual knowledge. The authors use instruction-following LLMs to automatically construct corpus with controlled factors above. The authors performed extensive analysis across different LM types and summarized several empirical findings, among which they show topic and style priors are easy to learn but factual knowledge is not.", "questions": "- There is a \"side note\" in page 7: \"When capacity is limited, the topic ratio and factual ratio simultaneously reduce on Pubmed in Figure 6.\" I did not see topic ratio reduces in Figure 6. Is this information supposed to be told by Figure 6?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The method how the analysis is performed is novel. Creating training and evaluation corpora with controlled differences (topics, style, factual knowledge) by prompting instruction following LLMs is interesting and inspiring.  \n- The analysis is extensive and is performed under various configurations (like the choice of LM, size of the training corpora)\n- The outcomes of analysis are interesting and relevant to future research that study lifelong learning of LMs.", "weaknesses": "- Although other configurations are very extensive, the choice of training and evaluation corpora and exclusively original or variants of PubMed and C4. \n- The three text-generating factors (styles, topics, facts) may not always be clearly separable of extensive enough in every corpora. The authors discussed this limitation in their limitation section.\n- Clarity issue: I feel the plots very hard to read because the captions are too generic and not self-contained. I suggest to briefly summarize the findings or implications in the captions.\n- Clarity issue: some legends in plots such as Figure 4 are not explained in text (e.g. readers may be confused about \"C4 -factuals\" before they associate them with \"C4-counterfactual\" in Table 1)\n- Though the authors pointed out the hardness of learning factual knowledge without learning style and topic bias, the authors' attempts failed to improve such performance at the end of Sec. 3. I suggest to provide some future directions about how the analysis will be beneficial the challenge of learning factual knowledge above.\n- The authors focused on evaluation of LM loss throughout the paper. I think this is fine for style and topics, but factual knowledge, evaluating LM loss is not clean enough because only a few tokens in a sentence are related to facts. The authors could create cloze-style  or question answering evaluation sets that focus exclusively on generation of factual knowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KX6fDquf1N", "reviewer": "Reviewer_QBAp", "summary": "The paper investigates the impact of finetuning language models on domain-specific texts and how it affects their general performance. The authors show that finetuning alters the model's preferences for topics and styles significantly, learning these features quickly and with minimal capacity. Factual knowledge, however, is acquired more slowly and requires greater capacity. The study's insights into language model learning dynamics could guide future enhancements in domain adaptation and help address the challenge of model forgetting during continuous learning.\n\nIn this study, the authors fine-tuned three models in increasing scales (GPT2-XL, LLaMa2 7B, and 13B) on PubMed abstracts with different scales of datasets up to 1M abstracts.", "questions": "* The basis of the method assumes that p(x)=p(topic,style,factual), but is there a justification to that decomposition? what about arithmetic? how does it fall to this decomposition?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Investigating the changes language models undergo after finetuning continues to be a highly relevant and evolving area of study, despite prior coverage in academic literature.\n\n2. The research offers key empirical insights into the differential impact of finetuning on language models, revealing a more pronounced effect on style and topic preferences compared to factual knowledge. These findings enhance our understanding of language model training dynamics and are instrumental in formulating more effective training methodologies.\n\n3. The researchers conducted extensive experiments on three language models of considerable size, particularly from an academic perspective.", "weaknesses": "* The assertion that each prediction by a language model can be broken down into components of writing style, topic, and factual knowledge requires further justification or explanation. The paper should present a stronger argument or provide additional evidence to substantiate this claim.\n\n* The primary message or conclusion of the paper is ambiguous. The authors need to clarify the central thesis to ensure that readers can grasp the main contribution of the work. what is the takeaway from this research? \n\n* While the prose is generally lucid, the paper's structure, particularly the introduction, could use refinement to enhance its readability and impact.\n\nFor improved clarity and presentation, the following suggestions are offered:\n1. The introduction would benefit from concrete examples illustrating the key domains of style, topic, and factual knowledge to help contextualize the subsequent findings.\n2. Details regarding the fine-tuning process are scant. Clarification of which specific models are assessed and the precise nature of the fine-tuning would provide a more robust understanding of the study's scope.\n3. The transition from discussing fine-tuning effects to probing methods in the third paragraph of the introduction is somewhat abrupt. A smoother segue that connects these topics would aid reader comprehension.\n4. In the fourth paragraph, where style and topic biases are introduced, it would be helpful to include examples or elaborate on what these biases entail to furnish readers with a clearer picture of these concepts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6MAbHbYsrO", "reviewer": "Reviewer_o2MC", "summary": "The paper studies the effects of finetuning on an LM. The authors disentangle the effects of the finetuning along three “dimensions”: topic, style, and factual knowledge. To achieve that, they leverage ChatGPT to generate a series of texts that are only different in one of those facets. Provided such texts, one can estimate log-likelihood ratio of different styles (for instance), by calculating differences of cross-entropies of the model on the texts.\nThe generated texts were verified by human judgements. \n\nThe experimental study is performed using two corpora (BioMed and C4) and three LMs (GPT-2 XL, LLaMa 2 - 7B & 13B). \n\nThe paper reports the following findings: (a) topic and style changing rapidly,(b) topic and style biases are independent, (c) topic and style require minimal capacity to be learned, in contrast to knowledge, (d) mixing in unbiased data only reduces the biases to a certain degree.", "questions": "* I wonder if there are other factorizations which can be studied in the same setup, apart from style/topic/knowldge?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* I find the topic of the investigation quite novel. I believe that the approach taken is original and innovative, in particular building a corpus that allows disentangling style/topic/factual knowledge. I also like the way LoRa was used to measure the capacity required for learning different facets.\n* The authors are sharing the data and code.\n* The reported experiments have provided some applicable insights, e.g. wrt the data mixing.", "weaknesses": "* Using synthetic data, generated by ChatGPT, might introduce some hidden biases. It is not given that the same findings could be found if we had natural data.\n* It is not clear if the same approach can be generalized to any other characteristics?\n\nTypos:\n* “by just changing the order of decomposition in 1” -> “...in Eq. 1”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)