Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition
Authors: Jiyeon Kim
Hyunji Lee
Hyowon Cho
Joel Jang
Hyeonbin Hwang
Seungpil Won
Youbin Ahn
Dohaeng Lee
Minjoon Seo
Venue: ICLR 2025 Oral (2025)
Keywords: knowledge entropy
knowledge acquisition and forgetting
evolving behavior during LLM pretraining
Primary area: foundation or frontier models, including LLMs
TL;DR: As pretraining progresses, models exhibit narrower integration of memory vectors, reflected by decreasing knowledge entropy, which hinders both knowledge acquisition and retention.

Abstract: In this work, we investigate how a model's tendency to broadly integrate its parametric knowledge evolves throughout pretraining, and how this behavior affects overall performance, particularly in terms of knowledge acquisition and forgetting. We introduce the concept of knowledge entropy, which quantifies the range of memory sources the model engages with; high knowledge entropy indicates that the model utilizes a wide range of memory sources, while low knowledge entropy suggests reliance on specific sources with greater certainty. Our analysis reveals a consistent decline in knowledge entropy as pretraining advances. We also find that the decline is closely associated with a reduction in the model's ability to acquire and retain knowledge, leading us to conclude that diminishing knowledge entropy (smaller number of active memory sources) impairs the model's knowledge acquisition and retention capabilities. We find further support for this by demonstrating that increasing the activity of inactive memory sources enhances the model's capacity for knowledge acquisition and retention.

Published as conference paper ICLR 2025
KNOWLEDGE ENTROPY DECAY DURING LANGUAGE MODEL PRETRAINING HINDERS NEW KNOWLEDGE ACQUISITION
Jiyeon Kim
Hyunji Leel
Hyowon Choo
Joel Jang
Hyeonbin Hwang!
Seungpil Won
Youbin Ahn
Dohaeng Lee
Minjoon Seo
KAIST AI
2 University of Washington
LG AI Research
{jiyeon.kim, hyunji.ame woonji.amy.... amy. minjoon)@kaist.ac ac.
ABSTRACT
In this work, we investigate how model's tendency to broadly integrate its parametric knowledge evolves throughout pretraining and how this behavior affects overall perfor- mance, particularly in terms of knowledge acquisition and forgetting. We introduce the concept of knowledge entropy, which quantifies the range of memory sources the model engages with; high knowledge entropy indicates that the model utilizes a wide range of memory sources, while low knowledge entropy suggests reliance on specific sources with greater certainty. Our analysis reveals consistent decline in knowledge entropy as pre- training advances. We also find that the decline is closely associated with a reduction in the model's ability to acquire and retain knowledge, leading us to conclude that di- minishing knowledge entropy (smaller number active memory sources) impairs the model's 's knowledge acquisition and retention capabilities. We find further support for this by demonstrating that increasing the activity inactive memory sources enhances the model's capacity for knowledge acquisition and retention
INTRODUCTION
Recent studies have analyzed how language models store world knowledge in their parameters and utilize this knowledge to generate responses during inference time (Geva et al. 2021 Dai et al.] [202aabb] Meng jet al. 2022 Yao al. 2024).. However, little known about how their behavior factual knowledge embedded in their parameters changes throughout the pretraining stage. In this work, we perform deep analysis of how a model's property broadly integrating diverse parametric knowledge evolves throughout pretraining and how these shifts affect overall performance, particularly in terms of knowledge acquisition and forgetting integration may explain why models in the later stages of pretraining encounter challenges in acquiring new knowledge (Dohare al. 2024, Jang et al., 2022 Chang al., 2024).
integrating various
continual learning setup. We hypothesize that this varying level of
We introduce knowledge entropy, which reflects how a language model integrates various knowledge sources, to investigate how this behavior evolves throughout pretraining. Recent studies have shown that feed-forward layers (FFNs) serve key-value memory (Geva al. 2022; 2021 Dai et al. 2022a). Building this research, as shown in Figure| we view the second projection matrix
as a memory, composed of memory
Denotes equal contribution
Codee in https::///// //gi httpps//ggihhb...////assss////// wtw.hhh..ommkkiissat////ooeeeeee

---

Published as conference paper ICLR 2025
FFN(X) -
[Continual Learning Performance]
Memory Vactors
Initaal
Mic
Final
Knowledge Entropy
Forgetting F((0)
Memory Coefficients
H(())
Acquisition A(()
Pretraining
Pretraining
Intial
Mid
Final
steps
K
steps
Figure I: Illustration our findings: distribution of memory coefficients C in feed-forward layers become sparser throughout pretraining, indicated by decrease in knowledge entropy H(00). This sparsity deteri- orates the model's knowledge acquisition A(d) and increases forgetting F(0) when conducting continual knowledge learning with models from different pretraining stages. Thereby, denoted by the star, when artificially increase the knowledge entropy the final stage model, both knowledge acquisition and retention increase.
vectors, which store the model's parametric knowledge, and view the first projection matrix K as generating coefficients C that determine how these memory vectors are combined. Knowledge entropy measures how sparsely these memory coefficients are distributed; high knowledge entropy indicates that the model tends to integrate broad range of memory vectors whereas the model with low knowledge entropy relies on specific memory vectors with high certainty. We analyze models at different stages of pretraining to investigate how knowledge entropy changes throughout pretraining. Our findings show that models in the later stages of pretraining tend to exhibit lower knowledge entropy, suggesting shift from utilizing a larger set of active memory vectors to a smaller, more focused set as pretraining progresses.
We hypothesize that changes in knowledge entropy would influence the model's behavior when encountering new knowledge. To test this, we conduct a thorough analysis of the model's 's ability to acquire new knowledge and retain existing knowledge in continual knowledge learning learnin@ssccnaario scenario Wu et al. 2023), starting from different stages of pretraining. This involves further training the pretrained model new-domain corpora using language modeling objective integrate results reveal a strong correlation between knowledge entropy and the model's 's ability to acquire and retain knowledge: both knowledge entropy and knowledge acquisition and retention decrease as the pretraining progresses.
target corpus (Jang al. 2022
knowledge. Our
We assume that this correlation arises because lower knowledge entropy indicates smaller of active memory vectors, leading to frequent overwriting of these memory vectors to store new knowledge. To test this assumption, we conduct experiments where we artificially increase the activity of previously inactive memory vectors, allowing the model to store new knowledge across broader range of memory vectors. Surprisingly, we observe that these modified models demonstrate improved knowledge acquisition and reduced forgetting compared to the original models when undergoing continual knowledge learning; though not to the same extent as the original pretrained model with equivalent knowledge entropy. Such a result bolsters hypothesis that having limited number of active memory vectors (low knowledge entropy) plays a critical role in explaining the degradation of the model's ability to acquire and retain knowledge as pretraining advances.
II this work, the "continual knowledge learning" and "continual learning" interchangeably.
2

---

Published as a conference paper at ICLR 2025
Overall, our findings reveal that as pretraining progresses, models exhibit narrower integration of memory vectors, reflected by decreasing knowledge entropy, which hinders both knowledge acquisition and retention. Models in the later stages of pretraining show low knowledge entropy, leading to poor knowledge acquisition and higher forgetting rates despite being trained on larger datasets. In contrast, early- stage models display high knowledge entropy, enabling better knowledge acquisition and retention; however, their performance is often limited by weaker language modeling capabilities. Thus, mid-stage models strike a balance, showing strong knowledge acquisition and retention along with overall performance, making them a practical choice for further training to incorporate new knowledge. To the best of our knowledge, this is the first work to analyze how a model's behavior in integrating various memory vectors changes across pretraining stages and the subsequent effects on performance when acquiring new knowledge in continual knowledge learning setup.
2 RELATED WORK
Dynamics of Knowledge in Language Models Recent studies have shown that language models embed world knowledge within their parameters and integrate this knowledge to generate responses (Yang) (004]
Petroni al., 2019; Wang et al.] (02I]. Thereby, various research efforts aim to understand these dynamics of knowledge in language models (how they learn, store, and engage their parametric knowledge) during
inference and training phases. Several studies have focused on investigating the inference process: Geva
ct al. (2023) analyzes the role of different layers in language models. Allen- -Zu & Li (2024b) demonstrates
that model parameters have limited knowledge capacity. Some studies suggest key-value memory (Geva
et al. 2021 Meng et al. 2022; Dai et al.) 2022a). Other research focuses on the pretraining phase. Liu et al.
(2021) ) studies the sequence that language models learn various types of knowledge, /lllenZZhu & Li| (2024a)
examines strategies to enhance knowledge storage and extraction. (Teehan al. analyzes internal structural changes. Sun & Dredze (2024) investigates the interaction between pretraining and finetuning. Chang et al.
(2024) analyzes patterns of knowledge acquisition specifically during the pretraining process, addressing the question of how language models acquire knowledge during pretraining. While their study shares similarities with ours in investigating knowledge acquisition behavior during LLM training, our work takes a different focus. We aim to understand why L.MMs encounter increasing difficulty in acquiring new knowledge as pretraining progresses, exploring the underlying reasons behind the challenges faced by later-stage models in learning new knowledge. To the best of our knowledge, our work is the first to explore how the behavior of language models in integrating their knowledge evolves throughout the pretraining phase, and to analyze how these changes affect model performance in terms of knowledge acquisition and forgetting in continual knowledge learning.
Entropy in Natural Language Processing In information theory, entropy quantifies the value of infor- mation, where predictable (certain) events have low entropy and unpredictable (uncertain) events have high entropy (Lairez 2022, Majenz 2018). In natural language processing (NLP), entropy is used in various ways to measure the certainty of language models. Yangl (2024) analyzes the entropy of model outputs based on input prompts. AAaauo et al. (2022) calculates the entropy of outputs at each layer to determine weight adjustments in a continual learning setup. Other papers focus on token probability entropy to understand the information required to predict the next word in sequence (Vazhentsev et al.) Geng et al.) 2024 Malinin & Gales 2021). Lower entropy in a model's predictions may indicate that the model has become more certain about its predictions based on training data. Additionally, Kumar& & Sarawagi (2019) measures entropy over the cross-attention layer to assess the uncertainty in the attention layer of encoder-decoder models. The entropy proposed in our paper, knowledge entropy. differs from previous work in that it focuses on the
Kumaa
'e define the final stage as the last stage of the pre-determined learning rate schedule, with the most decayed learning rate. The mid-stage refers to around 50% of the schedule, while the initial stage refers to around 20%.
3

---

Published as a conference paper ICLR 2025
entropy of model's parametric knowledge, assessing the uncertainty or variability in utilizing the knowledge encoded within the language model.
3 KNOWLEDGE ENTROPY
In this section, we introduce knowledge entropy (Section 3.1 to examine how broadly the model integrates its parametric knowledge and describe the experimental setup used to measure (Section 3.2). Next, in Section 3.3 we measure knowledge entropy various pretraining stages to analyze how the model's s knowledge integration behavior evolves over pretraining. In Section 3.4 we extend our investigation by exploring alternative definitions of entropy.
3.1 DEFINITION
In this work, we introduce a new concept, knowledge entropy, to analyze the scope of a model's access patterns to its parametric knowledge. Low knowledge entropy suggests that the model relies narrower set of specific knowledge sources with high certainty whereas high knowledge entropy indicates that the model integrates with diverse range of knowledge sources. Inspired by prior research that considers feed-forward layers (FFNs) key-value memory containing model's parametric memory (Geva et al.) [021] Dai et al.) 2022a Meng al., [022] Dong et al.] 2022), we consider the knowledge source
the memory vectors,
which is the second projection matrix of FFN. We measure how broadly the model integrates these memory vectors with memory coesficients, which are calculated by the first projection matrix and the activation function.
Geva et al. (222)) propose the concept of key-value memory, demonstrating that FFNs function similarly to the key-value neural memories (Sukhbaatar al. 2015). The feed-forward layer consists of two projection layers and activation in the middle:
F(x
(1)
where x E Rdd The first projection matrix (K E mx corresponds to the keys, and the second projection matrix (V E [R= represents the values, or the memories comprised of memory vectors. The output, Fx)( is a linear combination of the memory vectors Vi=1.... .m E Rd which are the rows of V. where the coefficientt C are determined by [(x K KT'), with / being a non-linear activation function such as ReLU. Previous studies have shown that various types of factual and linguistic knowledge are encoded within these memories (Dai et al. [222aa] Geva et al.) 2022. Meng et al. 2022 Dong et al.) (022). Thus, the final output is generated by combining the contributions of these memory vectors, where the memory coefficients determine the combination.
Thereby knowledge entropy, H(0),, calculated by the sum of laser-wise entropy H(O'), which is based on the average coefficient Ch Rm averaged across all tokens in dataset D, as described in Equation 2
Σ
(1 njj
prob
for i .......,7
Tn
2
j=1
1=
(2)
m prob ( (oroprrb((c)) (c));
H()))
H((0) Σ H(B))
WWe the terms "coefficient" and "memory coefficient" interchangeably.
4

---

Published as conference paper ICLR 2025
70
Attention Entropy
2.8
Attention Entropy
NTP Entropy
144.0
-8
78
297.6
2.7
NTP Entropy
Entropy
143.9
2977.4
Entropy
2.6
143.8
50
143.7
2.5
143.6
297.0
40
20
40 60
100
200
400
600
Rate of Pre-trained Step (%)
Pre-trained steps (k)
Figure 2: Entropy (y-axis) across different model states (x-axis) for OLMo IB and 7B. The x-axis represents the rate of the current step relative the last step (738k for 1B and 557k for 7B).
Figure 3: Entropy (y-axis) defined with Attention weight and next token prediction probability across different model states (x-axis) for OLMo 1B.
c represents the coefficient of the j-th token position of the n-th instance layer èt indicates the i-th element from Õ" Tn the sequence length the n-th instance in the dataset 2, m is the inner dimension of feed-forward layer, and L denotes the number of layers in the model.
3.2 EXPERIMENT SETUP
To conduct the experiment, we use the OLMo (Groeneveld et al.) 2024) models (1B and 7B), which are open-source large language models with intermediate pretraining checkpoints released, trained on the Dolma To measure knowledge entropy, we use a subset of Dolma, 2k instances that appear the first batch within the official pretraining data order to ensure that all models we are using have seen the corpus during pretraining step. Please note that the trend persists across other corpora as well (Figure7iin Appendix <..2p: however, since we are analyzing the model's behavior throughout training, we define knowledge entropy based calculations using the training dataset.
dataset (Soldaini et al. 2 S)
In the case of OLMo, the memory coefficient c(0) calculated as _.) abs(GGi)L where Xj the j-th token of input and SwiGLU (Shazeer 2020) is the activation function. We apply the absolute value since the SwiGLU allows negative values and the magnitude determines the contribution of the corresponding memory vector in the linear combination. Then, the absolute values are converted into a probability distribution. We also show experimentally that the trend persists with different choices activation functions. Further details regarding knowledge entropy can be found in Appendix A.2
3.3 FINAL MODELS TEND TO EXHIBIT LOWER KNOWLEDGE ENTROPY
Figure illustrates how knowledge entropy (y-axis) changes across different stages of pretraining (x-axis). The results show consistent decrease in knowledge entropy pretraining progresses in both 1B and 7B models. This trend suggests that models in the later stages of pretraining tend to engage with narrower range of memories, relying more heavily on specific memory vectors rather than accessing and integrating knowledge from a broader range of memories. A consistent reduction in knowledge entropy is observed across all layers, with the most significant reduction occurring in the last layer, which closely resembles the output distribution right before the token prediction (Figure 8 in Appendix).
'ee used OLMo and Dolma from the official repository
5

---

Published as a conference paper ICLR 2025
3.4 SIMILAR TRENDS ARE OBSERVED BY DIFFERENT DEFINITIONS OF ENTROPY
While our work defines knowledge entropy focusing on the feed-forward layer, previous studies have exam- ined entropy in different contexts, such the entropy of attention (Kumar & Sarawagi 2019) and the entropy of next-token prediction over the vocabulary space (Vazhentsev et al,; Malinin & Gales (021). To gain a more comprehensive understanding of the model's 's overall behavior, extend our analysis by exploring the entropy trends in both attention mechanisms and next-token prediction. The formula and details are provided in Appendix A.3)
Entropy of Attention Layers Following Kumar& & Sarawagi (011 we measure attention entropy to capture the degree of uncertainty in attention weights. is calculated as the sum of layer-wise entropy, where the layer-wise entropy measures the sparsity of the attention weights in each attention head. Thus, attention entropy reflects how much weight the model assigns to specific tokens with confidence when generating the next token given the input based on token relationships. Figure shows that attention entropy consistently decreases during pretraining, with sharp decline in the early stages followed by more gradual reduction. This trend suggests that the model learns to focus on contextually important tokens within the attention layer.
Entropy of Next Token Prediction Entropy can also be measured based on the probability distribution of next-token predictions over the vocabulary space (Vazhentsev et al.) Geng al. 2024 Malinin & Gales 2021). Figure 3 shows that the entropy of the next token prediction also consistently decreases throughout pretraining, reflecting the model's increasing certainty in its next token prediction.
4 KNOWLEDGE ACQUISITION AND FORGETTING
We hypothesize that the reduction of knowledge entropy as pretraining progresses impacts the model's s knowledge acquisition and forgetting as low knowledge entropy indicates sparse activation of memory vectors, thus the vectors are likely to be consistently overwritten when new knowledge is introduced. To test this hypothesis, measure knowledge acquisition and forgetting using checkpoints from different stages of pretraining in continual knowledge learning setup (Jang et al. 2022 Wu et al.; 2023), where further training is performed on new-domain corpora by next token prediction pretrained models. Section details the experimental setup and the metrics used. In Section 4.2 we present the results of knowledge acquisition and forgetting across various pretraining stages. Finally, Section 44. further explores whether a relationship exists between the two behaviors: activating the inactive memory vectors increases the knowledge acquisition ability.
inject new knowledge into the
4.1 EXPERIMENT SETUP
Model & Hyperparameters We experiment using intermediate checkpoints from OLM Hyp Hyperparame- ters are chosen following previous research on continual knowledge learning (Jang et al.) 2022 Kim et al. 2023) and we test various combinations to assess their generalizability. For batch size, test 128 and 2048, for learning rate, we experiment with le-4, 4e-4, and 1e-3. We also investigate the effect of training duration by comparing a single epoch to three epochs. Among these configurations, size of 128, learning rate of 4c-4, and single-epoch training as this setup most closely aligns with continual knowledge learning.
focus primarily
batch
'We select these intermediate checkpoints following Sun & Dredze 2024), which explores language model (OLMo) throughout pretraining. Available checkpoints can be found here
6

---

Published as a conference paper at ICLR 2025
Dataset We experiment on a subset of two datasets topics with abstracts, and C4 (Raffel et al. 2020). scage-scaleeeo corpus comprising diverse text data gathered from web pages. We use PubMed as the primary dataset as it contains more new knowledge, making it a better fit for our continual knowledge learning setup (Appendix [.I].. In addition to the dataset, we inject synthetic knowledge during training to assess the model's ability to acquire new information. Specifically, we utilizee FICTIONAL K KNOWLEDGE dataset (Chang et al.] 2024), which is designed to assess how well language models acquire factual knowledge during pretraining" | This dataset includes 130 paragraphs about fictional yet realistic entities and 1,950 probes where each paragraph contains 15 different probes. The passages are incorporated into the training batch 10 times during the continual knowledge learning. After training, we evaluate the models on evaluation probes of the Fictional Knowledge dataset to measure knowledge acquisition, and evaluate on six downstream tasks in zero-shot manner (Sun & Dredze 2024| Groeneveld
PubMed corpus of bio-medical and life science
et al. 2024) measure knowledge forgetting (SciQ (Welbl et al., 2017). Winogrande (Sakaguchi et al.
2021
PIQA (Bisk et al. 2020), OBQA (Mihaylov et al. 2018). Hel HellaS aawag Zellers et al. 2019), and ARC
Easy CClak al. 80)8). Detailed explanation is included in Appendix B.1
Metric Knowledge acquisition of language model 0 is measured with the probing performance on evaluation probes following Chang et al. (2024). When given the injected knowledge W, each instance w; in a corpus has a corresponding set of probes Pu:: containing 15 different probes. To measure how well the model recalls the injected knowledge, we compute the probe performance (C(O), the average log probability ((p;00) of the target span for each probe pt E Pu, across all instances in wy E W and calculate average; K(() = [W >n,ew [Pw,] En.p. E(p; 0). The knowledge acquisition metric A(0) is defined as the improvement rate of K(()) from Opr to dcc, where 0pT represents the model checkpoint from pretraining step, which serves as the starting point and Oct represents the model after continual knowledge learning. High A() indicates the model has learned new knowledge well.
To measure knowledge forgetting of a language model, we measure average performance over six down- stream tasks P0)). Knowledge forgefting F(0) is calculated by the reduction rate from OpT to 0CL Low F(0) indicates that the models have retained their existing knowledge. The equation and a detailed explanation are presented in Appendix B.2
4.2 KNOWLEDGE ACQUISITION AND RETENTION DECREASES ACROSS PRETRAINING STAGE
Figure 4a shows the performance of OLMo IB and 7B models from various stages of pretraining as an initial state. We observe that models in the final stages of pretraining struggle more with acquiring new knowledge A(#) and exhibit greater forgetting F(0). As shown in Figure 44b) continually training the models at the mid-point of the pretraining as the initial checkpoint tends to yield the best performance in knowledge probing and downstream tasks compared to both models from the initial and final stages of pretraining. While early-stage models demonstrate high knowledge acquisition with minimal forgetting, their overall performance is limited by weaker language modeling capabilities. Conversely, later-stage models, despite being trained on larger datasets, exhibit lower knowledge acquisition and higher rates of forgetting, resulting in lower overall performance compared to the mid-stage models. This aligns with previous research suggesting that model in the final stage of pretraining tends to struggle when learning new knowledge, showing a trade-off between plasticity and stability (Dohare et al. 2024 Biesialska et al. 2020 Jang et al.
We randomly sample 205k instances for each dataset. 'Datascts in huggingfacee
WW slightly modified the datasct our setting of which details are in Appendix B.1
For this experiment, we used PubMed as training corpus, but we also experiment with C4 as a training corpus, which presumably exhibits more similar distribution to the pretraining corpus, Dolma. The comparison of the two corpora
and the results can be found in Appendix B.I and B.4.2
7

---

Published as a conference paper ICLR 2025
20,0
Downstream perf P10) (%)
30
18
A(8)
Knowledge probe AC(6)
-0.25
8
Acquisition Al6o
78 A(B)
15.0
Forgetiing (
-0.26
62
27-
r111
12.5
0.27
18 K(8)
-60
24-
-0.28
70 K(0)
((8)
58
36
7.5
54
5.0
-0.30
-52
20
40
60
80
100
20
40
60 80
100
Rate Pre-trained Step (%)
Rate Pre-trained Step (%)
(b)
Figure 4: (a) Rates of knowledge acquisition A(0) and forgetting F(0) (b) End performance of knowledge probe (()) and downstream task P(0) for OLMo 1B (solid line) and 7B (dotted line) across different model states. The x-axis represents the ratio of the pretraining step of the initial model used in continual learning to
the final step. (738k for 1B and 557k for 7B)
2022p.
Therefore, we suggest that using a mid-stage checkpoint strikes good balance, making it a practical choice as an initial starting point for further training to inject new knowledge.
We consistently observe this pattern of later-stage models underperforming compared to earlier-stage models across various hyperparameter settings, including batch size, learning rate, training corpus, and the number of epochs. A detailed analysis of these results provided in Appendix [.44
4.3 RESUSCITATING INACTIVE MEMORY VECTORS INCREASES KNOWLEDGE ACQUISITION
We observe a strong correlation between the trend of knowledge entropy (Figure 2) and the model's 's ability acquire and retain knowledge (Figure 4ap.. We assume that the model's increasing reliance set of memory vectors (as indicated by decrease in knowledge entropy) leads to more frequent updates these vectors, making it difficult acquire new knowledge and resulting in a higher rate of forgetting. (The intuition behind this hypothesis can be found in Appendix I To test this assumption, we conduct experiments where we artificially increase the activity or resuscitate previously inactive memory vectors.
limited
To resuscitate inactive memory vectors, we modify the up-projection matrix K which engages with producing memory coefficients C (notations from Equation 2 Specifically, as shown Algorithm 1 we identify the lowest p% (resuscitation ratio) of memory coefficients and apply multiplier u to parameters in K that are associated with these p%. Multiplier can take any value; this experiment, we divide the mean coefficient value of each layer by the respective coefficient value è at each identified position i at layer I, and then multiply the result by an amplifying factor q. By varying the value of q, we control the degree of resuscitation applied to the p% low-activation coefficients, thereby influencing the magnitude of the average coefficient and the corresponding size of the parameter updates.
Figure 5a shows the knowledge acquisition and forgetting rates and Figure 5b presents the knowledge probe and downstream task performance after continual learning with various resuscitation configurations For the experiment, we fix p to 50 with varying and the OLMo checkpoint at the last step of pretraining. Results show that when q is set to 1 or greater, it generally yields better performance in both knowledge acquisition and retention compared the original model. In contrast, when g is to 0.5, which further reduces already
TTe Pearson correlation between knowledge entropy and knowledge acquisition is 0.94, and with forgetting, it is -0.96. Both correlations are statistically significant, with p-values of 6e-5 and le-5, respectively.
8

---

Published as conference paper ICLR 2025
70
-0.8
Downstrres cownstreaam rare per P101 T (%)
Knowledge probe (210)
A(d)
52.2
Acquisition A(0)
7(8)
Forgetting P16)
5221
-20
-0.30
40
K181 1
-1.9
-0.32
original
Q-0.5
q-1
d-2
q-3
original
gm0.5
q=1
qm2
qm3
Figure 5: (a) Rates of knowledge acquisition A(O) and forgetting F(0) (b) End performance of knowledge probe (C))) and downstream task P00) after training with varying the amplifying factor g (X-axis) while p fixed as 50.
inactive memory coefficients, both acquisition and retention decline suggesting that concentrating parameter updates more heavily on already active locations leads to sparser updates, ultimately impairing overall performance. These results suggest that having a narrower active memory vector (low knowledge entropy) fends to reduce the model's capacity to acquire new knowledge and increases knowledge forgetting.
Further experiments with fixed q and varying p show that increasing p to activate larger portion inactive parameters generally leads to improved performance. Detailed result of this configuration is in Appendix [B.6] We also analyze how the result changes when using models from different stages of pretraining as the original model. The trend holds consistently across different checkpoints of pretraining. However, the effect of the resuscitation becomes more pronounced as the original model progresses to later stages of pretraining. Detailed results are included in Appendix B.7
Our result indicates that resuscitating inactive memory vectors of final-stage models tends to enhance knowledge acquisition and overall performance compared to the unmodified final-stage model. However, we observe that the performance remains lower compared to models from the pretraining step with similar knowledge entropy, such the mid-stage model. This suggests that applying linear scaling subset of specific layers alone insufficient to induce fundamental behavioral changes in the model. In other words, restore model that has lost its plasticity (Dohare al., (024) its previous state, more fundamental and alternative approaches are required. Further exploring methods for effectively modifying the parameters would be an interesting direction for future work.
5 CONCLUSION
this work, we examine how large language models' ability broadly integrate their parametric knowledge (measured by knowledge entropy) changes throughout pretraining and how these changes affect knowledge acquisition and forgetting in continual learning setup. Our findings reveal strong correlation between knowledge entropy and the model's capacity to acquire and retain knowledge. Models in the final stages of pretraining tend exhibit narrower integration of memory vectors, leading to lower knowledge entropy, which negatively impacts both knowledge acquisition and retention. Interestingly. artificially increasing knowledge entropy by modifying the parameters of final-stage models tends to improve these capabilities. Based analysis, suggest that models from the mid-stage of pretraining offer good balance between knowledge acquisition, retention, and overall performance, making them a good choice for further training to introduce new knowledge.
9

---

Published as a conference paper ICLR 2025
Algorithm 1 Resuscitating Low Memory Coefficients
Require: C (average coefficients), p (resuscitation ratio), (amplifying factor), K (up-projection matrix)
Ensure: Scaled up-projection matrix K using computed multiplier
1: for each layer / in K do
2:
Extract average activations for layer I:
C = CI1
3: Compute the threshold t for the lowest p% activations:
percentile(C.,
4: Identify positions of values below the threshold
idx (C S n).nozeeoo( )
5: Compute the multiplier u for coefficients in layer
mean C*)
x
6: Apply scaling to the op-projection weights K2 at the identified positions:
Ki[idx;:] x= u(idx]
7: end for
6 LIMITATION & FUTURE WORK
Due to computational constraints, study measures knowledge acquisition and forgetting in continual learning setup. Future work could explore whether these behaviors also manifest during the pretraining phase. We focused on OL IB and 7B models, as they the only models that publicly provide inter- mediate pretraining checkpoints and demonstrate strong performance (Sun & Dredze 2024 Chang et al.) 2024). Extending this investigation to other models would be valuable direction for further research. Our resuscitation method, which arbitrarily modifies model parameters to test our hypothesis, showed promising results in improving knowledge acquisition and retention. However, performance tended to decline when resuscitating models at their initial or mid-stages. This suggests that more refined methods for resuscitating model parameters- ones that avoid random modification and preserve language modeling capabilitiees could yield better outcomes. Additionally, while we observed that models in the mid-stage of pretraining strike a good balance for further training on tasks that involve acquiring new knowledge, defining the mid-point precisely remains open question. In this study, approximated the mid-point as 50% of the learning rate schedule.
ACKNOWLEDGMENT
We thank Sohee Yang, Hoyeon Chang, Seongyun Lee, Seungone Kim, Doyoung Kim, and Hanseok Oh for helpful discussions and constructive feedback.
This work was supported by LG AI Research grant (Self-improving logical reasoning capabilities of LLMs, 2024, 50%) and the Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korca gover)men((MSIT) IRS-2024-00397966, Development of Cybersecurity Spe- cialized RAG-based sLLM Model for Suppressing Gen-AI Malfunctions and Construction of Publicly Demonstration Platform, 20%; No.2022-0-00113,, Developing a Sustainable Collaborative Multi-modal Lifelong Learning Framework, 20%; No.RS-2021-11222000 Artificial Intelligence Innovation Hub, 10%)
10

---

Published as a conference paper ICLR 2025
REFERENCES
AF Agarap. Deep learning using rectified linear units (relu), arXiv preprint arXiv:/803.08377 2018.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. In Forty-first International Conference on Machine Learning. 2024a. URL Utt.https://pperreview... net/
forum)id-55778rrbbc)
Zeyuan Allen-Zhre and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2404.005400 05500, 2024b.
Vladimir Araujo, Julio Hurtado, Alvaro Soto, and Marie-Francien Moens. Entropy- based plaltiiiiy for lifelong learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3721-3728, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Leurning, pp. 2397-2430. PMLR, 2023.
Magdalena Biesialska, Katarzyna Biesialska, and Marta Ruiz Costa-jussà, Continual lifelong learning in natu- ral language processing: survey. ArXiv, abs/2012.09823,, 2020. URL https://pp:: semanticscholar org 12:
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Hoyeon Chang. Jinho Park, Seonghyeon Ye, Sohee Yang. Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. How large language models acquire factual knowledge during pretraining? arXiv preprint arXiv:2406. 1813, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803. 05457, 2018.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493-8502, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10. 10.18653/1/20022.aca 581. (ttLhhttss///clanthoo hww.lloyy.oogg//// www.:oggg////... acn-long. 588
Damai Dai, Wen-Jie Jiang, Qingxiu Dong. Yajuan Lyu, Qiaoqiao She, and Zhifang Sui. Neural knowledge bank for pretrained transformers. Natural Language Processing and Chinese Computing, 2022b. URL
www.nttiammmatniiieec.a.eee http:://ppp..semnniiccccoooo.rrp hrw.aarrrrrr//CoossDDSS5
Shibhansh Dohare, J. Fernando Hermandez-Garcia. Qingfeng Lan, Parash Rahman, Ashique Rupam Mahmood, and Richard S. Sutton. Loss of plasticity in deep continual learning. Nature, 632:768 774, 2024. URL
https://api. www.seeeaannnnneeeeeeeeeeeeeeeee
orgg/CorppSI::259222 259251905
Qingxiu Dong, Damai Dai, Yifan Song. Jingjing Xu, Zhifang Sui, and Lei Li. Calibrating factual knowledge in pretrained language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguisties: EMNLP 2022, pp. 5937-5947, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. 438
finding--emnlp.438, URL http:pss:/acclanntooog.....o org.//y.

---

Published as a conference paper at ICLR 2025
Jiahui Geng, Fengyu Cai, Yuxia Wang. Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Pupers), pp. 6577-6595, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v 1/2024 naact-long.366
htt.ltttss///cc.atholoogyg org/ 2024. naacl- 366
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key- value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings the 2021 Conference Empirical Methods in Natural Language Processing, pp. 5484-5493 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18/653/vl httpw///211eccl 1 wwnlp-main.n4ee 446. httphhhts////ccanttiit org 2221 emnnp-mma main. 446
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 18.18653/v 1/w022.emmlp-mneee.. main....... /ttp::tts////caaattoto wwtp:/ooooooo.og/222//2/ 2022. emn emn lommlp--mn...
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216-12235, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.1886550
URL https: httpsss///aaaaanoooooorrrrog org/ wttgg//2...mmmmmmmaai..... mannlmmmmin..5111
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, al. Olmo: Accelerating the science of language models. ACL 2024, 2024.
Joel Jang, Seonghyeon Ye, Sohee Yang. Joongbo Shin, Janghoon Han, Gyeonghun KIM, Stanley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. In International Con- ference on Learning Representations, 2022. URL http/htts////peeeeeeee www.wewwwwwwweounneeeee weet/forum?iddvfsRRBBBB
Yujin Kim, Jachong Yoon, Seonghycon Ye, Sung Ju Hwang, and Se-young Yun. Carpe diem: on the evaluation of world knowledge in lifelong language models. NACCL 2024, 2023.
Aviral Kumar and Sunita Sarawagi. Calibration of encoder decoder models for neural machine translation. arXiv preprint 19X3v:1903.0000 00002 2019.
Didier Lairez. What entropy really is: the contribution of information theory. arXiv preprint 27X1:22204005477, 2022.
Zeyu Liu, Yizhong Wang. Jungo Kasai, Hannaneh Hajishirzi, and Noah A. Smith. Probing across time: What does RoBERTa know and when? In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 820-842, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1111865331/2221.1 tanddngss-emnl... URL
Christian Majenz. Entropy in quantum information theory-communication and cryptography. arXiv preprint /0t..//80000040 2018
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In Inter- national Conference on Learning Representations, 2021. URL IW5y-zb507m
12

---

Published as a conference paper ICLR 2025
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. http:httts////eeee/eeeeeee http:/eeee/ooruumi/idioaaaaaaeee h6WAS6EE
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2381 13911, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 18653/v1/D1$$ 1260. URL UBLlhttps///cllantto..o hrrpr//nntooooyyoo///1-------- org/D18-1260
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. Language models as knowledge bases? In Conference on Empirical Methods in Natural Language Processing, 2019. | //atp. semanticscholar. wrg. Lerg//CrpuSDDD-00 202539551
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, (1440)): 1-677 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9);99-106, 2021.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 2020.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. ACL 2024, 2024.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015.
Kaiser Sun and Mark Dredze. Amuro & char: Analyzing the relationship between pre-training and fine-tuning of large language models. arXiv preprint arXiv:2408.06666 2024.
Ryan Teehan. Miruma Clinciu, Oleg Serikov, Eliza Szczechla, Natasha Seelam, Shachar Mirkin, and Aaron Gokaslan. Emergent structures and training dynamics in large language models. ACL 2022. URL
https://aclanttiooogy..om org.22/2 http://222.iisccccceeeeeeee bigscience-1.11
Artem Vazhentsev, Akim Tsvigun, Roman Vashurin, Sergey Petrakov, Daniil Vasilev, Maxim Panov, Alexan- der Panchenko, and Artem Shelmanov. Efficient out-oin detection for sequence to sequence models. Findings of ACL 2023. URL httphhtss::aaaaattttloa//
org/ 2023. Findings-acl..
Cunxiang Wang, Pai Liu, and Yue Zhang. Can generative pre-trained language models serve as knowledge bases for closed-book QA? In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 Ith International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 3241-3251, Online, August 2021, Association for Computational Linguistics. doi: https://aclantholoogy www.orgg//2.... acl- long. 251
URL
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 94-106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 18653/v1/W 1/7744413,
orgg/17-441111
Yuhao WU, Tongjun Shi, Karthick Sharma, Chun Wei Seah, and Shuhao Zhang. Online continual knowledge learning for language models. arXiv preprint arXiv:2311.09622 2023.
13

---

Published as a conference paper ICLR 2025
Wenzhe Yang. Entropy, thermodynamies and the geometrization of the language model. arXiv preprint arXiv:2407.21092, 2024.
Yunzhi Yao, Ningyu Zhang, Zekun Xi, Meng Wang, Ziwen Xu, Shumin Deng, and Huajun Chen, Knowledge
circuits in pretrained transformers. ArXiv, abs/2405. 17969, 2024. https://pp/p semanticscholar org/CorpusID: 270068372
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Marquez (eds.), Proceedings the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19- 1472. URL https faeelannhoolgg..or///111---
A KNOWLEDGE ENTROPY
A.1 INTUITION BEHIND THE DEFINITION OF KNOWLEDGE ENTROPY
The mechanism behind the relationship between decreasing knowledge entropy and the ability to acquire and retain knowledge during pretraining is that the coefficients in the linear combination of memory vectors determine how the corresponding memory vectors updated. As defined in Equation T the output of the feed-forward layers (FFNs) is a linear combination of memory vectors Vi=1.-.. o E IR the row vectors of V E Rmxdd where the coefficients & are given by f(x ) In other words, FFN(2) C V1 C2 + Cm mm- Within given layer, since the operations beyond the FFNs and the input the FFNs remain consistent across all memory vectors, the coefficients act as scaling factors for the gradient by the chain rule. During training, the gradient OL for 1, m and j .., d can be decomposed Tvi,
as:
aL
AL OFFN(x)
dvij OFFN(x)
Ooi.
Here, 3FFN(2) is the same for all Vi, meaning that the relative magnitude of the gradient depends on aFrN(a) = G. Thus, larger coefficients result in proportionally larger gradients being applied to the dvL,
corresponding memory vectors, amplifying their updates during backpropagation.
As pretraining progresses, a spikier coefficient distribution-caaturee by decreasing knowledge en- tropy-implies that gradient updates become increasingly concentrated specific positions where the average coefficients are larger. This centralization can affect the model's ability to evenly utilize its memory capacity, thereby impacting knowledge acquisition and retention.
A.2 KNOWLEDGE ENTROPY
Does the choice of model change the trend? To assess the generalizability of the trend observed in Figure we conducted experiments the knowledge entropy trend using Pythia 1.4B model (Biderman et al. 2023). As shown in Figure 6. knowledge entropy measured with the Pythia model also tends to decrease pretraining progresses.
Does the choice of dataset change the trend? As expressed Equation knowledge entropy is dependent on the dataset D. We define D the dataset used during pretraining, knowledge entropy reflects how the model integrates the knowledge stored in its memory vectors, learned during pretraining. However, to further explore whether the choice of dataset influences the trend of knowledge entropy, we measure it using PubMed and C4. Figure / shows that the trend remains consistent regardless of the dataset used when calculating knowledge entropy.
14

---

Published as conference paper ICLR 2025
143.9
Know Knowledge ledge Entropy
143.7
Knowledde Entropy
C4
143.8
Pubmed
Dolma
143.7
143.6
143.6
143.5
143.5
143.4
143.3
300
700
Pre-trained steps (k)
Pre-trained steps (k)
Figure 6: Knowledge Entropy (y-axis) measured with Dolma on different pretraining stages Pythia 1.4B model.
Figure 7: Knowledge Entropy(y-axis) measured with Dolma, C4, Pubmed corpus across different pretrain- ing stages
Knowledge Entropy
9.00
Knowledge Entropy
8.98
9.27
8.96
9.24
8.94
9.21
8.92
18k
8.90
369
27Bk
9.12
2
5
25
30
Layer
Layer
(a)
(b)
Figure 8: Layer-wise Knowledge Entropy of OLMo-1B(a) and 7B(b)
Does the choice of activation function change the trend? We also explored an alternative where we do not take the absolute value of the SwiGLU output. Instead, following the ReLU function (Agarap) 2018). another widely used activation function, replaced all negative values with O. Figure shows that the trend remains consistent even under this modification.
C Rel.U(gate(xx)) up(x)),
Liyer-wise Knowledge Entropy Figure shows how knowledge entropy changes by layer during pretrain- ing. Knowledge entropy consistently decreases in every layer, with the most significant reduction occurring the last layer, which closely resembles the output distribution right before the token prediction. OLMo-7B model also shows similar trend 1B model.
15

---

Published as conference paper ICLR 2025
144.0
SwiGLU
3000
pubmed
ReLU
2500
143.8
Entropy
Frequency
2000
143.6
1500
143.4
143.2
100
200
300
400
500
600
700
Pre-trained Step
0
10
20
30
40
50
60
70
80
90
Average Perplexity Entities Instance
Figure Knowledge Entropy(y-axiss) when employ- ing original SwiGLU(solid line) and ReLU activa- tion(dotted line) across different pretraining stages
Figure 10: Histogram of average perplexity of enti- ties in C4 and Pubmed corpus
A.3 ENTROPY OF ATTENTION LAYERS
Inspired by previous research that emphasizes the attention layer's role attribute extraction (Geva et al. 2023), we also measure attention entropy H(O.n) similarly to Kumar & Sarawagi (2019 12 Attention weights, which are the output of softmax normalization after the key-query-value operation, can be interpreted as weight assigned the previous tokens. As the attention weight for each token position in each attention head to I), calculating entropy follows the normal entropy formula. Then,
forms a probability Distributionssumnin layer-wise entropy H(O is averaged over token position and attention heads and attention entropy H0.mn) is the of layer-wise entropy H H(().)). Following the notations from Geva al. (2023). attention entropy calculated as:
(0
Σ
A
(og(AA).)))
for = 1.,,..., Tr and j 1., Tr
is
T
1( h.
H (0tit) (0ant)
H(O')))
H(O
H(Oma)
Σ
H((0)))
D
T
N
1=1
where A A,1. xTT represents the attention weights the /-th attention head in layer / for the n-th instance, T, is the sequence length of the n-th instance in the training dataset D, N denotes the number of attention heads, and L denotes the number of layers in the model.
A.4 ENTROPY OF NEXT TOKEN PREDICTION
The entropy of next token prediction (Vazhentsev et al. Geng et al. 2024 Malinin & Gales 2021 is defined H(O)))) ! p; log(p:), where p, represents the probability of the i-th token. This value is then averaged over the sequence length (T.) and the dataset size ((D)).
Kumar & Sarawagi (0119) measures entropy using attention weights from crossattentiin in an encoder-decoder architecture. However, employ decoder-only model, modify the equation attention weights from self-attention.
16

---

To measure the forgetting rate, we evaluate 6 Published as a conference paper at ICLR 2025
B KNOWLEDGE ACQUISITION AND FORGETTING
B.1 DATASETS
Training Dataset for Continual Knowledge Learning In this section, we share a brief description of the datasets used. For continual knowledge learning, we experiment with PubMed and C4. The PubMed dataset consists of biomedical literature abstracts from the PubMed database, containing articles across a wide range of topies in medicine and biology. The C4 (Colossal Clean Crawled Corpus) dataset (Raffel et al. 2020) is a large-scale, preprocessed collection of text scraped from the web, designed to be a clean and diverse representation of natural language.
To compare the distribution of C4, PubMed, and Dolma, we evaluate the average perplexity of entities for C4 and PubMed, as shown in Figure 10 On the x-axis, we plot the range of an average perplexity of instances, while the y-axis represents the number of instances. We randomly sample 10,000 instances from each corpus, extract entities using GPT-4o, and calculate perplexity with the last checkpoint of OLMo. The perplexity values from the last checkpoint of OLMo indicate how likely these entities are to appear in the pretraining corpus, Dolma. The results reveal that PubMed exhibits a broad distribution of perplexity, with a higher number of instances having high perplexity values. In contrast, C4 shows tendency towards lower perplexity, suggesting that the distribution of entities in PubMed differs from that in Dolma, while the distribution in C4 tends to be more similar to Dolma.
Evaluation Dataset to measure Knowledge Acquisition To measure knowledge acquisition of language model, we use the fictional knowledge dataset (Chang et al. 2024). which is designed assess how well LLMs acquire factual knowledge during pretraining. This dataset includes 130 paragraphs [3 presented in a Wikipedia-style format with fictional yet realistic entities (injected knowledge) and 1,950 probes which are cloze-task-style sentences to query the information within the corpus. The final span of each probe, referred to as the target span, is used to evaluate the model's 's prediction probability, which serves as a measure of knowledge acquisition performance.
In Chang et al. (2024), the probes are divided into three levels of difficulty, with five sentences created for each level. This results in 15 probes per corpus. The difficulty levels are follows: 1) Memorization probes directly ask about sentences explicitly present in the fictional corpus. 2) Semantic generalization probes are paraphrased versions of the memorization probes test the model's understanding of meaning beyond surface forms. 3) Compositional generalization probes are designed to assess whether the model can integrate multiple pieces of knowledge from the fictional corpus. The injected knowledge is incorporated into the training corpus during continual learning, with updates occurring every 160 steps.
Following Chang et al. (2024). we divide the 130 corpora into two settings: paraphrase and once. In the paraphrase setting, 70 instances are each paraphrased 10 times. For every 160 steps, one paraphrased version of an instance added the training corpus, repeating this process 10 time [Ta] In the once setting, each instance is presented only once throughout the entire continual learning process. The 60 instances are divided into 10 groups, with 6 instances added every 160 steps.
Evaluation Dataset to measure Knowledge Forgetting downstream datasets.
Whiie Chang al. (2024) utilizes 120 paragraphs, comprising 40 for paraphrase setup, 40 for duplicate, and 40 for once, we utilized 130 paragraphs, the whole original data. We scaled up the number of paragraphs with paraphrases to 70, utilizing GPT-4 following Chang et al. (2024)
"We maintain updates every 160 sTops(T0 steps when batch size is 2048) because our total training duration is 1,600 steps(100 steps), and we repeat the dataset injection process 10 times.
17

---

Published as conference paper ICLR 2025
SCIQ: multiple-choice question-answering dataset consisting of over 13,000 science exam-style questions, covering subjects such as physics, chemistry, biology, and earth science
WINOGRANDE: large-scale benchmark designed to test commonsense reasoning in natural language understanding
commonsense reasoning about everyday physical interactions such how to perform tasks involving physical actions
PIQA:
multiple-choice question-answering answering benchmark designed assess model's ability answer elementary-level science questions.
OBQA:
HELLASWAG: a large-scale benchmark for commonsense reasoning, focusing on selecting the most plausible continuation of given narrative scene.
ARC EASY: multiple-choice science questions typically answered by students in elementary and middle school.
B.2 METRIC
In this section, we share a detailed description of how we evaluate knowledge acquisition and knowledge forgetting.
Knowledge Acquisition Given language model 0, Opr represents the model extracted from a pretraining step and serves as the initial point for continual learning, and fcL represents the model after it. The acquisition metric A(0) for the model 0 is defined Equation [4] When given a corpus set of once setting W onces each instance in corpus w; has a corresponding set of probes Pw., which contains 15 different probes. To calculate the performance of the once setting, Konce (0). we compute the average log probability ((pç0) of the target span for each probe E Pw, across all instances in W; E W and sum these averages. The same calculation is performed for the paraphrase setting. The total performance K((0) is calculated by the weighted average of Kounce (0) and (0) Finally, the acquisition metric A(l) is defined as the improvement rate in performance K((f) from the initial model state Opr final model state Ict-
Konce (0)
Σ
E(p: 0);
Kpara (0)
Σ
Σ
E(p:; 0)
W once
P
W
P
10, E W
PiEPec
para
No Wiaa
PIEP=
W
oncel
Krnnce (0)
+ paral
x
Kparaa
K((0c..)
K((OpT)
KC(O)
A(0)
W Wonce
+
W
para
K((Pr)
(4)
Knowledge Forgetting The forgetting metric F(8) is calculated the average performance degradation from the initial model 4er to the final model 0cL, across six downstream tasks T: SciQ (Welbl et al. 2017). Winograde (Sakaguchi et al.] (021), PIQA (Bisk al. 2020), OBQA (Mihaylov et al. 2018). HellaSwag (Zellers et al. 2019), and ARC Easy (Clark et al. 2018).
F(O) P(fc)) P(Grr) P(Dpr)
P(0)
Σ T((0); 1ETT
(5)
T
18

---

Published as conference paper ICLR 2025
-020
0.28
Downstree Nownstream eam pert P10) 7 (9%)
40
660
Knowledge probe xC(0)
Acquisition A(B1 T
30
-0250
-0.30
62
Parashraze JIBI
Knowledg obe
C. A30
-00
Parashrose x16
-0.32
Once Jid
-0.300
PT K(0m)
-0325
-0.34
0.350
0.36
54
0.375
0-
52
-0400
100
200
300
400
500
600
700
100
200
300
400
500
600
700
Pre-trained Step (k)
Pre-trained Step (k)
Figure 11: Rates knowledge acquisition A()) and final performance of knowledge probe (C(0) according to the number of injection. Paraphrase prompts (solid line) were injected 10 times and Once (dotted line) prompt only once throughout con- tinual training.
knowledge probe KC(0) and downstream task P(0) of initial model (dotted line) and model after continual train- ing (solid line) in baseline setup. The x-axis repre- sents the initial model state used for continual train- ing.
Figure 12: End performance
20
0.28
Downstrea eettreamm perf P(8) (%)
Pubmed .181-1
18
Knowledge probe x300)
-57
22
C4 FOBt
Puteed 1100
14
Forgetiing P(8)
-0.29
Puomed KIB)
Puomed
56
Acquisition
12
+
55
-0.30
10
50
18
59
-0.31
16
52
100
200
300
500
600
100
200
300
400
5000
600
700
Pre-trained Step
Pre-trained Step
(a)
(b)
Figure 13: (a) Rates of knowledge acquisition A(0) and forgetting F(0) (b) End performance knowledge probe IC)0) and downstream task (() of Pubmed(solid line) and C4(dotted line) corpus across different model states. The x-axis represents the initial model state used for continual training.
B.3 FREQUENCY OF KNOWLEDGE INJECTIONS
We divide the experiment into two settings: the once setting, where knowledge is injected single time, and the paraphrase setting, where knowledge injected times using ten paraphrased paragraphs. Figure presents knowledge acquisition results based the frequency of injections. Knowledge acquisition and final performance generally follow similar trends in both settings, with models the later stages of pretraining showing the lowest performance. However, the performance and acquisition rate of once setting lag behind those of paraphrase setting. Also, notably, for models in the final stage of pretraining, the acquisition rate in the once setting negative. This indicates that the log probability of the injected knowledge decreased, preventing successful incorporation the new knowledge. In other words, even the knowledge injected during continual learning is subject to forgetting throughout the continual learning process.
19

---

Published as a conference paper ICLR 2025
B.4
KNOWLEDGE ACQUISITION & FORGETTING
B.4.1 BASELINE SETUP
Our base experiments are conducted using hyperparameter configuration most closely aligned with continual knowledge learning studies, specifically with a batch size of 128, a learning rate of 4e-4, while training single-epoch of PubMed corpus. We use adamW optimizer (3 0.9, 0.95, weight decay= 0.1), cosine LR scheduler with warmup=0.05, and set maximum sequence length as 1024. We randomly selected 204,800 instances from the PubMed and C4 datasets and matched the sequence length to 1,024 tokens by concatenating instances. This resulted in a training dataset consisting of approximately 210 million tokens.
As analyzed in Section 4.2 final performance generally deteriorates later-stage models are utilized the initial model. Figure 12 illustrates the model's initial performance before continual learning, as well as its performance afterward. Models in the later stages of pretraining exhibit superior language modeling abilities before continual learning, as evidenced by the lower log probability for the newly injected knowledge (dotted line). However, after continual learning, their performance deteriorates compared models from the earlier stages of pretraining (solid line). Similarly, the downstream task performance of the later-stage models was better initially, but after continual learning, their performance declined more than that of the earlier-stage models.
B.4.2 VARIOUS SETTINGS
We observed consistently, even with altered hyperparameter settings, that models in the later stages of pretraining struggle to learn new knowledge and retain existing knowledge.
Training Dataset for Continual Knowledge Learning To investigate how the type of new knowledge affects performance, further conducted experiments using the C4 corpus, which has distribution more similar to the pretraining corpus, Dolma, compared to PubMed. Figure [egure[3afindin [ndindicates that the gap in acquisition rate between later-stage models and initial-stage models is larger when the new knowledge distribution differs significantly from the pretraining corpus: models trained with PubMed (-6.4%p) exhibit a more pronounced gap compared those trained with C4 (-3.4%p).
All models, regardless of their pretraining stage, tend to perform better when continually pretrained with PubMed compared C4. We hypothesize that this is because PubMed's different distribution from the pretraining corpus encourages the model to learn more new knowledge, enhancing its ability to acquire new information. However, the rate of improvement varies by model state. Later-stage models tend show similar performance regardless of the type of new knowledge, suggesting limit in their learning capacity. In contrast, initial-stage models exhibit a stronger ability to acquire knowledge when trained on a corpus with a different distribution, such as PubMed, demonstrating their greater adaptability in learning new and diverse information.
Model Size To test the universal deterioration of knowledge acquisition and retention capabilities, we also experimented with the OLMo 7B model Groeneveld et al. (2024). In Figure 4a the 7B model exhibits a clear trend of diminishing knowledge acquisition A(0) capabilities as pretraining progresses. This decline is accompanied by an increase in forgetting F(8), indicating that the model struggles to retain previously learned information as new data introduced.
Batch Size In Table 1 line (b), when the batch size is large, the influence of individual data points the model update decreases, resulting in a lower acquisition rate, while forgetting is less pronounced. In the later stages of training, however, if sufficient learning rate warmup is not provided, the model appears to collapse.
20

---

Published as conference paper ICLR 2025
A(0)
F(0)
Pretraining Step of 0
118k
369k
554k
738k
118k
369k
554k
738k
(a) Baseline
25.2
24.8
21.4
18.8
10.5
12.9
17.0
19.5
(b) BS :128 + 2048
8.9
7.8
1.0
-92.2
2.2
3.5
5.5
44.9
(c) Ir: 4c-4 + Ic-3
17.3
15.7
10.7
5.0
16.7
20.3
23.0
23.5
(d) Ir: 4e-4 + le-4
21.6
23.0
23.7
22.6
3.2
4.6
(c) ep: 3
28.8
28.0
25.9
24.5
14.8
17.4
20.1
23.1
Table 1: Knowledge Acquisition A(])[%) and Forgetting F(0)[%) across different Continual Learning hyperparameters.
(C)
P(0) T
Pretraining Step of
118k
369k
554k
738k
118k
369k
554k
738k
(a) Baseline
-0.281
-0.280
-0.293
-0.301
54.0
54.6
52.8
52.0
(b) BS :128 + 2048
-0.342
-0.344
-0.368
-0.714
59.0
60.5
60.1
35.6
(c) Ir: 4c-4 + lc-3
-0.310
-0,314
-0.332
-0.353
50,3
49.9
49.0
49,4
(d) Ir 4e-4 - le-4
-0.294
-0,287
-0,284
-0.287
58.9
60.7
60.6
59.9
(e) ep: 3
-0.267
-0.268
-0.276
-0,280
51.4
51.7
50.8
49.7
Table 2: Probe Performance KC(#) and average performance on six downstream task P(0) across different Continual Learning hyperparameters.
Learning Rate In Table line (c), when the learning rate is increased, only is the acquisition rate lower but forgetting becomes more pronounced. Conversely, when the learning rate is very small (in line (d)), the models in the late stage show improved acquisition performance with the least gap compared with initial models, but still perform worse than the mid-stage model.
Epoch In Table [ line (c), as the number of epochs increases, leading to more repetitions, acquisition improves with the cost of more forgetting.
B.5 TRAINING DYNAMICS OF PYTHIA
1.4B and OLMo IB. We observe that the overall training dynamics of Pythia and OLMo are similar. However, Pythia tends to conclude early in training, using only 10% of training tokens compared OLMo. As result, Pythia struggles to capture the full dynamics of the language model during pretraining. This is because models in the early stage of training are still influenced by a large portion of randomly initialized parameters, which leads different behavior compared to models that have undergone more training steps and reached stable state. For OLMo, based on our analysis, we assume the model reaches a stable point around 118k steps (15% the training dataset). However, Pythia's final step occurs before this stable point, causing its trend resemble that of OLMo but deviate from the trend observed in Section 4 where knowledge acquisition improves until the stable point and then decreases. We hypothesize that previous studies using intermediate checkpoints during
Figure 14 shows the training dynamics of Pythia Biderman al. 2023
21

---

Published as conference paper ICLR 2025
Olmo
144.1
Knowledge Acquisition
Knowledge Entropy
144.0
24
143.9
143.8
22
143.7
20
143.6
0
1000
2000
3000
Pythia
Knowlede Acquisition
Knowledge Entropy
143.7
26
143.6
25
143.5
143.4
1000
2000
3000
Tokens (M)
Figure 14: Knowledge Entropy and Rates of knowledge acquisition A(0) of OLMo 1B and Pythia 1.4B model across different pretraining stages
19.5
52.4
(%)
Knowledge probe 7
Acquisition A(8)
Ate)
-0.99
Downstren eam P10)
710)
Forgetiing 71
52.2
18.0
17.5
52.1
17.0
-0.31
16.5
(68)
51.9
-0.32
original
P-30
p-50
p-70
p-90
p=
p=70
(a)
(b)
Figure (a) Rates of knowledge acquisition A(0) and forgetting F(0) (b) End performance of knowledge probe IC(0) and downstream task P0) after training with varying (x-axis) while is fixed as 1.
pretraining (Chang al. 2024 Sun & Dredze 2024) also only measured with OLMo without Pythia for the
same reason.
B.6 RESUSCITATION EXPERIMENT: VARYING p WHILE FIXING !
Figure 15 shows the overall performance when was fixed at 1, meaning that the lowest p% of coefficients were scaled to converge toward the layer's mean. It is shown that increasing p activate a larger portion of
22

---

Published as conference paper ICLR 2025
20
-0.26
Downstr cownstream eam per P10 7 (%)
Acquisition A(8)
10
Knowledge probe (C10)
0.27
54.0
Forgetting F(0)
-0.28
53.5
resusclation .(9)
-10
-0.29
original 718)
resuscitation K(d)
-0.30
original K2() 1
restacitation Ptel
-0.31
52.0
100
200
300
400
500
600
700
Pre-trained Step (k)
Pre traineridd Step
(a)
(b)
Figure 16: (a) Rates of knowledge acquisition A(0) and forgetting F(0) (b) End performance of knowledge probe (C)) and downstream task P(0) of original continual learning (dotted line) and resuscitation (solid line) method where p fixed as and q as 2, across different model states. The x-axis represents the initial model state used for continual training.
20
$4.5
Downstream perf P(O) (96)
Knowledge probe A318)
Acquisition .le)
Forgetting 710)
5.0
369k .188)
554k .18)
369K 710)
-0.29
369k
368K
53355
330
554k
554k
40
-50
original 0-0.5
q=1
Q-2
q-3
original
g-0.5
q-1
q-2
qw3
(a)
(b)
Figure 17: (a) Rates of knowledge acquisition A(0) and forgetting F(0) (b) End performance of knowledge probe IC(0) and downstream task P()) of model from 369k steps (solid line) in pretraining schedule and 554k (dotted line) when varying the amplifying factor y (x-axis) while p is fixed as 50.
inactive parameters generally led improved performance. However, interestingly, setting p too high(e.g.. at 90) negatively impacted performance, likely due to the unintended effect reducing the parameters in already active regions.
B.7 RESUSCITATION EXPERIMENT ACROSS PRETRAINING STEPS
Figure 16a illustrates the overall performance when & was fixed to and to 50, across different pretraining stages of the original model. The effect of the resuscitation method becomes more pronounced as the original model progresses later stages of pretraining, as indicated by the transition from the dotted line to the solid line. We hypothesize that this because late-stage models tend to rely on a smaller subset of memory sources and thus benefit from broader scope of activation enabled by the resuscitation method. As shown in Figure [6b] end performance deteriorates when the beginning model is initial (118k) and mid (369k) stage model, indicating that resuscitation may impair performance when the model's knowledge entropy is not sufficiently low. This trend the resuscitation showing a more positive effect for models the later stage of pretraining also be in Figure 17 which shows the results when varying 9 while fixing p at 50:
23

---

Published as conference paper ICLR 2025
-0.99
Downstream per M10) (%%
50
Knowledge probe A0090
52.1
Forgetting 710)
40
Original
Acquisition
Attn Resuschation
FFN Resusdtation
51.8
0.31
20
Ofiginal Atn Resusctation PFN Resuuctittion
51.7
-30
1.5
2.0
3.0
1.5
2.0
3.0
(a)
(b)
Figure 18: (a) Rates of knowledge acquisition A(0) and forgetting F(0) (b) End performance of knowledge probe IC(O) and downstream task P(0) when resuscitation is applied at attention layers, with the x-axis representing temperature. The dotted line indicates performance when resuscitation is applied the feed- forward layer with p=50, q=2. The dashed line represents the original performance without any resuscitation.
performance deteriorates when running continual learning on the model from 369k, while improvement of performance with larger g is observed when the model is from 554k.
B.8 RESUSCITATION EXPERIMENT ACROSS ATTENTION LAYERS
In Figure we observe that as pretraining progresses, not only knowledge entropy but also attention entropy decreases. We explore whether extending the resuscitation method the attention layers could similarly improve knowledge acquisition and retention. For the feed-forward layer, we can adjust knowledge entropy by tuning the parameters positions with low activation values. However, since attention entropy reflects the probability distribution over token positions, the same resuscitation method cannot be directly applied to the attention layers. artificially increase attention entropy, we employed temperature scaling on the softmax function in the attention calculation, which reduces the sparsity of the probability distribution.
We experimented with temperature values ranging from 1.5 to 3.0. Figure [8 shows that resuscitation the feed-forward layer consistently results in the highest knowledge acquisition rate and generally leads to the best knowledge retention. This trend observed in both knowledge probe results and downstream performance. These findings suggest that knowledge entropy plays critical role in influencing both knowledge acquisition and forgetting, thereby driving the observed differences in performance.
24


Reviews:
[{"review_id": "zSzoWL6Fkc", "reviewer": "Reviewer_hpJw", "summary": "The authors identified a phenomenon where language models use less diverse set of value vectors (i.e., the parametric memory store) to learn knowledge. The authors called the distribution \"knowledge entropy\" and hypothesize that the decrease in knowledge entropy hinders the models' ability to acquire and retain new knowledge. The authors found that the knowledge entropy decreases as pretraining progresses. \n\nThe authors further test the progression of knowledge acquisition (using fictional knowledge probes) and forgetting (using common knowledge datasets) with intervention on the up-projection matrix. The author resusitate the inactive memory vectors by finding the least active $p\\%$ of the memory vectors and multiply the up-projection matrix by the ratio $q$ at the corresponding index. The authors show that this reduce forgetting and increase retention of new knowledge (I have questions regarding this -- see the question section).", "questions": "In L209, the negative value is often understood to remove/erase the information from the residual stream. If this flexibility is removed from the model, it could affect the model's plasticity. It would be useful to see a more flexible gating function. The entropy can still be calculated using the absolute values of the coefficients (and does not need to change the model arch and training).\n\nWhat does section 3.4 do to support the use of knowledge entropy? I understand the concept of attention entropy and next token entropy but how does that connect to the choice of using knowledge entropy? In fact, if all these entropy measures decrease over time, how can we justify that knowledge entropy is the main reason for the degraded plasticity even if it shows high correlation? Section 4.3 provides interventional evidence but similar intervention should be done with, say, attention entropy to ensure this is not the source for the lowered plasticity. I.e., if making attention entropy less peaky also increase retention and mitigates forgetting, then the role of knowledge entropy is not unique as other statistics reflect the same phenomenon. Would be nice to see more interventional evidence showing that knowledge entropy is the main contributor to the plasticity issue.\n\nFigure 5 (b): seems that downstream performance decreases as the $q$ value gets larger, which is the opposite of knowledge retention? And the knowledge probing result in the same plot doesn't show improvement over lower $q$ values. Am I missing something or does it show a different trend than Figure 5 (a)?\n\nI'm happy to increase my scores if the above questions can be addressed.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper investigates an important problem of knowledge acquisition and retention in LLMs. The angle is simple, clear, and easy to validate. The idea is not particularly high on the originality end but the identified \"variable\" (i.e., the knowledge entropy) has not been used to understand forgetting/retention as far as I know.", "weaknesses": "**Main Issues**\n\nThe main concern is it's unclear how causal the knowledge entropy is to the phenomenon of knowledge forgetting/retention. The paper provided mostly correlational evidence. Please see the Questions section for more concrete critique and suggestions.\n\n**Presentation Issues**\n\nQuite a bit of notation overloading:\n1. c is used for the memory coefficient in equation 2 and used again in section 4.1 L303 to represent knowledge.\n2. $m$ is used for index in equation 2 and then as multiplier in algorithm 1\nL320: missing space between sentences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fOz8EAu2XF", "reviewer": "Reviewer_zgzo", "summary": "This work introduces the concept of \"knowledge entropy\" to analyze how language models integrate their stored knowledge throughout the pretraining process. By viewing feed-forward layers as key-value memory systems, the researchers measure how broadly or narrowly models access their stored knowledge by calculating the entropy of memory coefficients. Their key finding is that as pretraining progresses, models exhibit decreasing knowledge entropy, meaning they tend to rely on a narrower set of memory vectors rather than broadly integrating knowledge from many sources. This narrowing correlates strongly with decreased ability to acquire new knowledge and retain existing knowledge during continual learning, an interesting empirical finding.", "questions": "1. Why do you think that knowledge entropy correlates with decreased ability to acquire new knowledge? \n\n2. Is the resuscitation method intended solely as a test for your main finding, or do you also envision a path in which this could practically be useful for increasing plasticity of models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The concept of \"knowledge entropy\" introduced in this paper is novel and very interesting. (1) Entropy based metrics have been used in previous works before. (2) Recent works have gradually built the view of feed-forward layers as key-value memory. This paper has a refreshed view, combining these 2 insights to give a new metric. \n\nThe \"resuscitation\" method, by manipulating parameters, presents somewhat a direct test for the authors' main claim that decreased knowlege entropy gives rise to decreased ability to learn new things.", "weaknesses": "This work only tested on OLMo models (1B and 7B), as acknowledged by authors. The authors state their reasons for this as due to the fact that OLMo provide intermediate checkpoints. \n\nHowever the Pythia models, too, offer intermediate checkpoints. If time permits, the authors should endeavor to check their results on these models too, in order to ascertain the generality of their claims and whether it holds beyond a single model family. \n\nIt would also be useful if the authors provide some discussion regarding mechanisms behind their main finding (knowledge entropy correlates with decreased ability to acquire new knowledge). It does not need to be theorem based, which is a difficult endeavor. Empirical findings would help, to delve at least a bit deeper into this phenomenon and why it might occur.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I5boikCr6n", "reviewer": "Reviewer_EdjN", "summary": "This paper investigates how knowledge acquisition and forgetting evolves through an LLM's pretraining. Building on the work of Chang et al 2024 and Geva et al 2021 and 2022, the authors introduce a concept of knowledge entropy and show that low entropy, which corresponds to poor knowledge acquisition and high forgetting rate, occurs in late stages of pretraining while high entropy occurs for initial stages of pretraining.", "questions": "Please address the questions about bias, and also about how knowledge entropy links to intuitions about types of memories.  Finally please answer the questions about probes and knowledge acquisition.\nCould you more clearly articulate how this paper builds on chang et al with respect to the part on knoweldge acquisition and forgetting?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This work exposes an important feature of pretraining and contributes valuable insights to LLM pretraining.  \nWhile the definition of knowledge entropy to this reviewer is not completely clear, it is an important feature of it that it patterns with other notions of entropy in models like attention entropy or next token prediction entropy.   The work on knowledge acquisition and forgetting is potentially very important for how we view model training.\n\n\nThe empirical results are solid and convincing to this reviewer.", "weaknesses": "This paper has the potential to make substantial contributions to the NLP and ML communities.  But I believe it still could benefit from further work.\n\nThe talk of broad and particular memory vectors in the introduction is a bit unclear.\n\nIn calculating entropy, the authors assume the feed forward network has the form\nFFN(x) = f(x · K^T ) · V\nbut this leaves out important bias terms that might perhaps affect the calculations.  Can the authors say more about this?\n\n\nThere are some important connections with Chang et al 2024 that need to be brought out more.  As it stands, it seems as though much of the heavy lifting is done in Chang et al, and this paper provides an incremental improvement to that paper.  There needs to be a discussion of how this paper goes beyond the previous one.  \n\nAlso it is important to not oversell the information contained in the so called memory vectors.  In particular, given the experiments with human interpreters given the strings that affect these memory vectors the most,  what we get are suggestive interpretations of what's stored, that is suggestive to those particular human interpreters.  As far as I know Chang et al shows mostly  it's not clear that the model can make use of them in the way they are interpreted.\n\nWhile it's pretty clear intuitively what attention entropy or next token prediction entropy are, it's less clear at least to this reviewer what the intuition is behind defining entropy in terms of the coefficients of the outputs of a given FFN layer in the model.   How does the knowledge entropy so defined link up with the suggestion that \"models in the later stage of pretraining tends to engage with a narrower\nrange of memories, relying more heavily on specific memory vectors rather than accessing and integrating\nknowledge from a broader range of memories\"?\n\nAnother element that needs a bit more work is the discussion of knowledge acquisition and forgetting.  This relies on work from Chang et al. but to be more self-contained the authors could explain in a bit more detail what the probes are doing (and why, for instance, 15 different probes?), or rather summarize Chang et al's contributions more clearly with respect to the contributions of the present work.\n\nThat part as it stands leaves this reader wanting more information.  For instance, what is the target span for the probes?  Also is \\theta_{pt} unique? Is there only one such check point and what about \\theta_{cl}?  Is \\theta_{cl} when the continual learning stops, if it does, or what?  Is  \\theta_{cl} then the model after pre training?   There is more material in the appendix on this but it would help the reader's understanding of this important concept of knowledge acquisition and knowledge loss  to move some of that discussion into the body of the paper.\n\nSome minor stylistic corrections that will render the paper more readable:\n\nWorks ---> work   (`work' is still a mass known in English, as far as I know)\ncoefficient of j-th token  ---> coefficient of the  j-th token\n\ninto probability distribution --> into a probability distribution\n\nWe also experiment that the trend persists ---> We also show experimentally that the trend persists\n\nwe inject FICTIONAL KNOWLEDGE dataset  ---> we inject elements of the FICTIONAL KNOWLEDGE dataset??\n\nof target span  ---> of the target span\n\nEquation and detailed explanation is\npresented  ---> please fix the English\n\nHoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon\nSeo. How do large language models acquire factual knowledge during pretraining? arXiv preprint\narXiv:2406.11813, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eVkJPjNmtD", "reviewer": "Reviewer_yokT", "summary": "The paper introduces knowledge entropy as a novel concept for the quantification of model representation across Large Language Models. The authors show that as the training of a large model progresses, the knowledge tends to concentrate in a smaller set of \"memory vectors\", making the process of knowledge acquisition difficult (decrease in plasticity), and the model more prone to forgetting prior knowledge as memory vectors are forced to be constantly rewritten. Experimental results show that it is indeed the case that models in the later stages of their training are more resilient to learning new knowledge, and tend to easily forget the knowledge that they already have. In addition, by resuscitating the inactive vectors, the authors force a later-stage model to ameliorate both the aforementioned concerns in the continual learning process.\n\nOverall, I think this is a good paper with interesting results that can have further applications for the community. Some concerns and questions still remain as outlined below, but I am willing to increase my score once they are addressed.", "questions": "- Given that consistent behavior is observed across all layers, do you think we can use only a subset of layers to determine the model's proneness to learning new knowledge? For example, only focusing on layers that mainly encode semantic knowledge as shown in the literature. \n- Related to the question above, is the same trend observed across different forms of knowledge and is there a difference between various knowledge forms (syntactic knowledge versus semantic knowledge for example)? \n- Given the minor negative effects of resuscitation of inactive vectors, do you think this is a viable method for adaptation in continual learning frameworks in generals to minimize forgetting and increasing knowledge acquisition? More specifically, what negative effects can this artificial resuscitation have?\n- In Figure 5(b), what is the reason behind the decrease in the downstream task performance while a drastic decrease in forgetting is also observed? Shouldn't a low forgetting rate lead to better general-purpose performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed framework of categorizing knowledge retention in terms of the model's knowledge entropy is interesting albeit somewhat expected as it is generally understood that moving away from established weight matrices becomes more difficult as the training progresses.\n- The concept of entropy in this case is both well-defined and intuitive, essentially showing how variant the coefficients are with respect to each input token given an instance.\n- The experimental results confirm the theoretical hypothesis, strongly indicating that knowledge entropy is associated with both learning and forgetting.", "weaknesses": "- The usage of the term \"entropy\" in this case can be confusing in my opinion. Usually, this term implies some form of \"surprise\" with respect to information. However, in this case, it seems to refer to the idea of source broadness. \n- I think this is a paper that could benefit greatly from further evaluation with respect to knowledge entropy. Although it is expected that the same behavior is observed across models, I would like to know if different architectures affect the observed behavior and if different training time design choices can regulate the knowledge entropy on a case-by-case basis.\n- Further discussion is required regarding the correspondence of knowledge entropy and forgetting. Rather than being a direct consequence of knowledge entropy, can it be the case that the reason behind sharper declines in knowledge retention is due to the model's richer knowledge representation between concepts at later stages of the training and thus being more prone to knowledge loss when small changes are made? More specifically, forgetting at a higher rate at later stages can be expected simply due to having more knowledge encoded in comparison to having very little knowledge in earlier stages. \n- Minor grammatical errors that can be fixed in the camera-ready version.\n- I believe that Algorithm 1 can be rewritten to better relay the information contained. Namely, by giving a better explanation of each operation (especially with respect to $t$).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)