Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisition

## Abstract

Abstract In this work, we investigate how a model’s tendency to broadly integrate its parametric knowledge evolves throughout pretraining, and how this behavior affects overall performance, particularly in terms of knowledge acquisition and forgetting.
We introduce the concept of knowledge entropy , which quantifies the range of memory sources the model engages with; high knowledge entropy indicates that the model utilizes a wide range of memory sources, while low knowledge entropy suggests reliance on specific sources with greater certainty. Our analysis reveals a consistent decline in knowledge entropy as pretraining advances. We also find that the decline is closely associated with a reduction in the model’s ability to acquire and retain knowledge, leading us to conclude that diminishing knowledge entropy (smaller number of active memory sources) impairs the model’s knowledge acquisition and retention capabilities.
We find further support for this by demonstrating that increasing the activity of inactive memory sources enhances the model’s capacity for knowledge acquisition and retention 1 1 1 Code in https://github.com/kaistAI/Knowledge-Entropy.git .

## 1 Introduction

Recent works have analyzed how language models store world knowledge in their parameters and utilize this knowledge to generate responses during inference time (Geva et al., ; Dai et al., ; ; Meng et al., ; Yao et al., ).
However, little is known about how their behavior of integrating various factual knowledge embedded in their parameters changes throughout the pretraining stage.
In this work, we perform deep analysis into how a model’s property of broadly integrating diverse parametric knowledge evolves throughout pretraining and how these shifts affect overall performance, particularly in terms of knowledge acquisition and forgetting in a continual learning setup.
We hypothesize that this varying level of integration may explain why models in the later stage of pretraining encounter challenges in acquiring new knowledge (Dohare et al., ; Jang et al., ; Chang et al., ).

Figure: Figure 1: Illustration of our findings: distribution of memory coefficients $\bar{C}$ in feed-forward layers become sparser throughout pretraining, as indicated by a decrease in knowledge entropy $\mathcal{H}(\theta)$. This sparsity deteriorates the model’s knowledge acquisition $\mathcal{A}(\theta)$ and increases forgetting $\mathcal{F}(\theta)$ when conducting continual knowledge learning with models from different pretraining stages. Thereby, as denoted by the star, when we artificially increase the knowledge entropy of the final stage model, both knowledge acquisition and retention increase.
Refer to caption: x1.png

We introduce knowledge entropy, which reflects how a language model integrates various knowledge sources, to investigate how this behavior evolves throughout pretraining.
Recent studies have shown that feed-forward layers (FFNs) serve as a key-value memory (Geva et al., ; ; Dai et al., ). Building on this research, as shown in Figure 1, we view the second projection matrix $V$ as a memory, composed of memory vectors, which store the model’s parametric knowledge, and view the first projection matrix $K$ as generating coefficients $\bar{C}$ that determine how these memory vectors are combined.
Knowledge entropy measures how sparsely these memory coefficients are distributed; high knowledge entropy indicates that the model tends to integrate broad memory vectors whereas the model with low knowledge entropy uses certain memory vector with high certainty.
We analyze models at different stages of pretraining to investigate how knowledge entropy changes throughout pretraining.
Our findings show that models in the later stages of pretraining tend to exhibit lower knowledge entropy, suggesting a shift from utilizing a larger set of active memory vectors to a smaller, more focused set as pretraining progresses.

We hypothesize that knowledge entropy change would influences the model’s behavior when encountering new knowledge.
To test this, we conduct a thorough analysis of the model’s ability to acquire new knowledge and retain existing knowledge in a continual knowledge learning(^2^22In this work, we use the term “continual knowledge learning” and “continual learning” interchangeably.) scenario on a target corpora (Jang et al., ; Wu et al., ), starting from different stages of pretraining.
This involves further training the pretrained model on new-domain corpora using a language modeling objective to integrate new knowledge.
Results show a strong correlation between knowledge entropy and the model’s ability to acquire and retain knowledge: both knowledge entropy and knowledge acquisition and retention decrease as the pretraining progresses.

We assume that this correlation arises because lower knowledge entropy indicates a smaller set of active memory vectors, which leads to frequent overwriting on these memory vectors to store new knowledge.
To test this assumption, we conduct experiments where we artificially increase the activity of previously inactive memory vectors, allowing the model to store new knowledge across a broader range of memory vectors.
Surprisingly, we observe that these modified models demonstrate improved knowledge acquisition and reduced forgetting compared to the original models when undergo continual knowledge learning; though not to the same extent as in the original pretrained model with equivalent knowledge entropy.
Such a result bolsters our hypothesis that having a limited number of active memory vectors (low knowledge entropy) plays a critical part in explaining the degradation of the model’s ability to acquire and retain knowledge as pretraining advances.

Overall, our findings reveal that as pretraining progresses, models exhibit narrower integration of memory vectors, reflected by decreasing knowledge entropy, which hinders both knowledge acquisition and retention.
Models in the later stages of pretraining(^3^33We define the final stage as the last stage of the pre-determined learning rate schedule, with the most decayed learning rate. The mid-stage refers to around 50% of the schedule, while the initial stage refers to around 20%.), show low knowledge entropy, leading to poor knowledge acquisition and higher forgetting rates despite being trained on larger datasets.
In contrast, early-stage models display high knowledge entropy, allowing them better in knowledge acquisition and retention; however, their performance is often limited by weaker language modeling capabilities.
Thus, mid-stage models strike a balance, showing strong knowledge acquisition and retention along with overall performance, making them a practical choice for further training to inject new knowledge.
To the best of our knowledge, this is the first work to analyze how a model’s behavior in integrating various memory vectors changes across pretraining stages and the subsequent effects on performance when acquiring new knowledge in a continual knowledge learning setup.

## 2 Related Works

##### Dynamics of Knowledge in Language Models

Recent studies have shown that language models embed world knowledge within their parameters and integrate this knowledge to generate responses (Yang, ; Petroni et al., ; Wang et al., ).
Thereby, various research aims to understand these dynamics of knowledge in language models (how they learn, store, and engage their parametric knowledge) during inference and training phases.
Several works have focused on investigating the inference process:
Geva et al. () analyzes the role of different layers in language models.
Allen-Zhu & Li () demonstrates that model parameters have a limited knowledge capacity.
Some studies suggest key-value memory (Geva et al., ; Meng et al., ; Dai et al., ).
Other research focuses on the pretraining phase.
Chang et al. () explores how language models acquire knowledge.
Liu et al. () studies the sequence that language models learn various types of knowledge.
Allen-Zhu & Li () examines strategies to enhance knowledge storage and extraction.
analyzes internal structural changes. Sun & Dredze () investigates the interaction between pretraining and finetuning.
Our work is the first to explore how the behavior of language models in integrating their knowledge evolves throughout the pretraining phase, and to analyze how these changes affect model performance in terms of knowledge acquisition and forgetting in continual knowledge learning.

##### Entropy in Natural Language Processing

In information theory, entropy quantifies the value of information, where predictable (certain) events have low entropy and unpredictable (uncertain) events have high entropy (Lairez, ; Majenz, ). In natural language processing (NLP), entropy is used in various ways to measure the certainty of language models.
Yang () analyzes the entropy of model outputs based on input prompts. Araujo et al. () calculates the entropy of outputs at each layer to determine weight adjustments in a continual learning setup.
Other works focus on token probability entropy to understand the information required to predict the next word in a sequence (; Geng et al., ; Malinin & Gales, ). Lower entropy in a model’s predictions may indicate that the model has become more certain about its predictions based on training data. Additionally, Kumar & Sarawagi () measures entropy over the cross-attention layer to assess the uncertainty in the attention layer of encoder-decoder models.
The entropy proposed in our paper, knowledge entropy, differs from previous works in that it focuses on the entropy of a model’s parametric knowledge, assessing the uncertainty or variability in utilizing the knowledge encoded within the language model.

## 3 Knowledge Entropy

In this section, we introduce knowledge entropy (Section 3.1) to examine how broadly the model integrates its parametric knowledge and an experiment setup to measure it (Section 3.2).
Next, in Section 3.3, we measure knowledge entropy at various pretraining stages to analyze how the model’s knowledge integration behavior evolves over pretraining.
In Section 3.4, we extend our investigation by exploring alternative definitions of entropy.

### 3.1 Definition

In this work, we introduce a new concept, knowledge entropy, to analyze the scope of a model’s access patterns to its parametric knowledge.
Low knowledge entropy suggests that the model relies more on a narrower set of specific knowledge sources with high certainty whereas high knowledge entropy indicates that the model integrates with a diverse range of knowledge sources.
Inspired by prior works that consider feed-forward layers (FFNs) as key-value memory containing model’s parametric memory (Geva et al., ; Dai et al., ; Meng et al., ; Dong et al., ), we consider the knowledge source to be the memory vectors, which is the second projection matrix of FFN, and measurement of how broadly the model integrates as memory coefficients, calculated by the first projection matrix.

Geva et al. () propose the concept of key-value memory, demonstrating that FFNs function similarly to the key-value neural memories (Sukhbaatar et al., ).
The feed-forward layer consists of two projection layers and activation in the middle:

$$ $FFN(\mathbf{x})=f(\mathbf{x}\cdot K^{T})\cdot V$ (1) $$

where $\mathbf{x}\in\mathbb{R}^{d}$.
The first projection matrix ($K\in\mathbb{R}^{m\times d}$) corresponds to the keys, and the second projection matrix ($V\in\mathbb{R}^{m\times d}$) represents the values, or the memories comprised of memory vectors.
The output, $FFN(\mathbf{x})$, is a linear combination of the memory vectors $\mathbf{v}_{i=1,\cdots,m}\in\mathbb{R}^{d}$ which are the rows of $V$, where the coefficients(^4^44We use the terms “coefficient” and “memory coefficient” interchangeably.) $C$ are determined by $f(\mathbf{x}\cdot K^{T})$, with $f$ being a non-linear activation function such as ReLU.
Previous studies have shown that various types of factual and linguistic knowledge are encoded within these memories (Dai et al., ; Geva et al., ; Meng et al., ; Dong et al., ).
Thus, the final output is generated by combining the contributions of these memory vectors, where the memory coefficients determine the combination.

Thereby knowledge entropy, $\mathcal{H}(\theta)$, is calculated by the sum of layer-wise entropy $\mathcal{H}(\theta^{l})$, which is based on the average coefficient $\bar{C}^{l}\in\mathbb{R}^{m}$ averaged across all tokens in dataset $\mathcal{D}$, as described in Equation 2.

$$ $\displaystyle\bar{C}^{l}=\frac{1}{|\mathcal{D}|}\sum_{n=1}^{|\mathcal{D}|} \left(\frac{1}{T_{n}}\sum_{j=1}^{T_{n}}C^{(l)}_{n,j}\right);\qquad\text{prob}( \bar{c}^{l}_{i})=\frac{\bar{c}^{l}_{i}}{\sum_{k=1}^{m}\bar{c}^{l}_{k}},\quad \text{for }i=1,2,\ldots,m$ (2) $\displaystyle\mathcal{H}(\theta^{l})=-\sum_{i=1}^{m}\text{prob}(\bar{c}^{l}_{i })\cdot\log(\text{prob}(\bar{c}^{l}_{i}));\qquad\mathcal{H}(\theta)=\sum_{l=1} ^{L}\mathcal{H}(\theta^{l})$ $$

$C^{(l)}_{n,j}$ represents the coefficient of $j$-th token position of $n$-th instance at layer $l$, $\bar{c}^{l}_{i}$ indicates an $i$-th element from $\bar{C}^{l}$, $T_{n}$ is the sequence length of the $n$-th instance in the dataset $\mathcal{D}$, $m$ is the inner dimension of feed-forward layer, and $L$ denotes the number of layers in the model.

### 3.2 Experiment Setup

To conduct the experiment, we use the OLMo (Groeneveld et al., ) models (1B and 7B), which are open-source large language models with intermediate pretraining checkpoints released, trained on the Dolma dataset (Soldaini et al., )(^5^55We used OLMo and Dolma from [the official repository](https://github.com/allenai/OLMo/tree/main)). To measure knowledge entropy, we use a subset of Dolma, 2k instances that appear in the first batch within the official pretraining data order to ensure that all models we are using have seen the corpus during pretraining step. Please note that the trend persists across other corpora as well (Figure 7 in Appendix A.1); however, since we are analyzing the model’s behavior throughout training, we define knowledge entropy based on calculations using the training dataset.

In the case of OLMo, the memory coefficient $C^{(l)}_{n,j}$ is calculated as $C^{(l)}_{n,j}=\text{abs}(\text{SwiGLU}(\mathbf{x_{j}}))$ where $\mathbf{x_{j}}$ is the j-th token of input $\mathbf{x}$ and SwiGLU (Shazeer, ) is the activation function.
We apply the absolute value since the SwiGLU allows negative values and the magnitude determines the contribution of the corresponding memory vector in the linear combination. Then, the absolute values are converted into probability distribution.
We also experiment that the trend persists with different choices of activation functions. Further details regarding knowledge entropy can be found in Appendix A.1.

### 3.3 Final models tend to exhibit lower knowledge entropy

Figure: Figure 2: Entropy (y-axis) across different model states (x-axis) for OLMo 1B and 7B. The x-axis represents the rate of the current step relative to the last step (738k for 1B and 557k for 7B).
Refer to caption: x2.png

Figure 3 illustrates how knowledge entropy (y-axis) changes across different stages of pretraining (x-axis). The results show a consistent decrease in knowledge entropy as pretraining progresses in both 1B and 7B models. This trend suggests that models in the later stage of pretraining tends to engage with a narrower range of memories, relying more heavily on specific memory vectors rather than accessing and integrating knowledge from a broader range of memories. Consistent reduction in knowledge entropy is observed across all layers, with the most significant reduction occuring in the last layer, which closely resembles the output distribution right before the token prediction (Figure  8 in Appendix).

### 3.4 Similar Trends are observed by different definitions of Entropy

While our work defines knowledge entropy focusing on the feed-forward layer, previous studies have examined entropy in different contexts, such as the entropy of attention (Kumar & Sarawagi, ) and the entropy of next token prediction over the vocabulary space (; Malinin & Gales, ). To gain a more comprehensive understanding of the model’s overall behavior, we extend our analysis by exploring the entropy trends in both attention mechanisms and next token prediction. The formula and details are in Appendix A.2)

##### Entropy of Attention Layers

Following Kumar & Sarawagi (), we measure attention entropy to capture the degree of uncertainty in attention weight.
It is calculated as the sum of layer-wise entropy, where the layer-wise entropy measures the sparsity of the attention weight in each attention head.
Thus, attention entropy reflects how much weight the model assigns to specific tokens with confidence when generating the next token given the input based on token relationships. Figure 3 shows that attention entropy consistently decreases during pretraining, with a sharp decline in the early stages followed by a more gradual reduction. This trend suggests that the model learns to focus on contextually important tokens within the attention layer.

##### Entropy of Next Token Prediction

Entropy can also be measured based on the probability distribution of the next token prediction over the vocabulary space (; Geng et al., ; Malinin & Gales, ).
Figure 3 shows that the entropy of the next token prediction also consistently decreases throughout pretraining, reflecting the model’s increasing certainty in its next token prediction.

## 4 Knowledge Acquisition and Forgetting

We hypothesize that the reduction of knowledge entropy as pretraining progresses impacts the model’s knowledge acquisition and forgetting as low knowledge entropy indicates sparse activation of memory vectors, thus the vectors are likely to be consistently overwritten when new knowledge is introduced.
To test this hypothesis, we measure knowledge acquisition and forgetting using checkpoints from different stages of pretraining in a continual knowledge learning setup (Jang et al., ; Wu et al., ), where further training is performed on new-domain corpora by next token prediction to inject new knowledge to the pretrained models.
Section 4.1 details the experimental setup and the metrics used.
In Section 4.2, we present the results of knowledge acquisition and forgetting across various pretraining stages.
Section 4.3 further explores whether a relationship exists between the two behaviors: activating the inactive memory vectors increases the knowledge acquisition ability.

### 4.1 Experiment Setup

##### Model & Hyperparameters

We experiment using intermediate checkpoints from OLMo(^6^66We select these intermediate checkpoints following Sun & Dredze (), which explores language model (OLMo) throughout pretraining. Available checkpoints can be found [here](https://github.com/allenai/OLMo/blob/main/checkpoints/official/OLMo-1B.csv).).
Hyperparameters are chosen following previous research on continual knowledge learning (Jang et al., ; Kim et al., ) and we test various combinations to assess their generalizability.
For batch size, we test 128 and 2048; for learning rate, we experiment with 1e-4, 4e-4, and 1e-3.
We also investigate the effect of training duration by comparing a single epoch to three epochs. Among these configurations, we focus primarily on batch size 128, learning rate 4e-4, and single epoch training as this setup most closely aligns with continual knowledge learning.

##### Dataset

We experiment on a subset of two datasets(^7^77We randomly sample 205k instances for each dataset.): PubMed (^8^88Datasets in [huggingface](https://huggingface.co/datasets/ncbi/pubmed)), a corpus of bio-medical and life science topics with abstracts, and C4 (Raffel et al., ), a large-scale corpus comprising diverse text data gathered from web pages.
We use PubMed as the primary dataset as it contains more new knowledge, making it a better fit for our continual knowledge learning setup (Appendix B.1).
In addition to the dataset, we inject synthetic knowledge during training to assess the model’s ability to acquire new information.
Specifically, we inject Fictional Knowledge dataset (Chang et al., ), which is designed to assess how well language models acquire factual knowledge during pretraining(^9^99We slightly modified the dataset to our setting of which details are in Appendix B.1).
This dataset includes 130 paragraphs about fictional yet realistic entities and 1,950 probes where each paragraphs contain 15 different probes.
The passage is incorporated into the training batch 10 times during the continual knowledge learning.
After training, we evaluate the models on evaluation probes of the Fictional Knowledge dataset to measure knowledge acquisition, and evaluate on six downstream tasks in zero-shot manner (Sun & Dredze, ; Groeneveld et al., ) to measure knowledge forgetting (SciQ (Welbl et al., ), Winogrande (Sakaguchi et al., ), PIQA (Bisk et al., ), OBQA (Mihaylov et al., ), HellaSwag (Zellers et al., ), and ARC Easy (Clark et al., )). Detailed explanation is included in Appendix B.1.

##### Metric

Knowledge acquisition of a language model $\theta$ is measured with the probing performance on evaluation probes following Chang et al. ().
When given the injected knowledge $C$, each instance $c_{i}$ in a corpus has a corresponding set of probes $P_{c_{i}}$, containing 15 different probes. To measure how well the model recalls the injected knowledge, we compute the probe performance $\mathcal{K}(\theta)$, the average log probability $\ell$($p_{i}$;$\theta$) of target span for each probe $p_{i}\in P_{c_{i}}$ across all instances in $c_{i}\in C$ and calculate average; $\mathcal{K}(\theta)=\frac{1}{|C|}\sum_{c_{i}\in C}\frac{1}{|P_{c_{i}}|}\sum_{p
_{i}\in P_{c_{i}}}\ell(p_{i};\theta)$.
The knowledge acquisition metric $\mathcal{A}(\theta)$ is defined as the improvement rate of $\mathcal{K}(\theta)$ from $\theta_{\text{PT}}$ to $\theta_{\text{CL}}$, where $\theta_{\text{PT}}$ represents the model checkpoint from a pretraining step, which serves as the starting point and $\theta_{\text{CL}}$ represents the model after continual knowledge learning. High $\mathcal{A}(\theta)$ indicates the model has learned new knowledge well.

To measure knowledge forgetting of a language model, we measure average performance over six downstream tasks $\mathcal{P}(\theta)$. Knowledge forgetting $\mathcal{F}(\theta)$ is calculated by the reduction rate from $\theta_{\text{PT}}$ to $\theta_{\text{CL}}$. Low $\mathcal{F}(\theta)$ indicates the models have retained its existing knowledge.
Equation and detailed explanation is presented in Appendix B.2.

Figure: (a)
Refer to caption: x4.png

### 4.2 Knowledge Acquisition and Retention Decreases across Pretraining Stage

Figure 4 shows the performance of OLMo 1B and 7B models (^10^1010For this experiment, we used PubMed as a training corpus, but we also experiment with C4 as a training corpus, which presumably exhibits more similar distribution to pretraining corpus, Dolma. The comparison of the two corpus and the results can be found in Appendix B.1 and B.4.2) corpus from various stages of pretraining as an initial state. We observe that models in the final stage of pretraining struggle more with acquiring new knowledge $\mathcal{A}(\theta)$ and exhibit greater forgetting $\mathcal{F}(\theta)$.
As shown in Figure 4, continually training the models at the mid-point of the pretraining as the initial checkpoint tends to yield the best performance in knowledge probing and downstream tasks compared to both models from the initial and final stage of pretraining.
While early-stage models demonstrate high knowledge acquisition with minimal forgetting, their overall performance is limited by weaker language modeling capabilities. Conversely, later-stage models, despite being trained on larger datasets, exhibit lower knowledge acquisition and higher rates of forgetting, resulting in lower overall performance compared to the mid-stage models.
This aligns with previous research suggesting that a model in the final stage of pretraining tends to struggle when learning new knowledge, showing a tradeoff between plasticity and stability (Dohare et al., ; Biesialska et al., ; Jang et al., ).
Therefore, we suggest that using a mid-stage checkpoint strikes a good balance, making it a practical choice as an initial starting point for further training to inject new knowledge.

We consistently observe this pattern of later-stage models underperforming compared to earlier-stage models across various hyperparameter settings, including batch size, learning rate, training corpus, and the number of epochs. A detailed analysis of these results is provided in Appendix B.4.

### 4.3 Resuscitating inactive memory vectors increases knowledge acquisition

We observe a strong correlation(^11^1111The Pearson correlation between knowledge entropy and knowledge acquisition is 0.94, and with forgetting, it is -0.96. Both correlations are statistically significant, with p-values of 6e-5 and 1e-5, respectively.) between the trend of knowledge entropy (Figure 3) and the model’s ability to acquire and retain knowledge (Figure 4).
We assume that the model’s increasing reliance on a limited set of memory vectors (decrease in knowledge entropy) leads to more frequent updates to these vectors, making it difficult to acquire new knowledge and resulting in a higher rate of forgetting.
To test this assumption, we conduct experiments where we artificially increase the activity or resuscitate previously inactive memory vectors.

Figure: (a)
Refer to caption: x6.png

To resuscitate inactive memory vectors, we modify the up-projection matrix $K$ which engages with producing memory coefficients $\bar{C}$ (notations from Equation 2).
Specifically, as shown in Algorithm 1, we identify the lowest $p\%$ (resuscitation ratio) of memory coefficients and apply a multiplier $m$ to parameters in $K$ that are associated with these $p\%$.
Multiplier $m$ can be any numbers; in this experiment, we divide the mean coefficient value of each layer by the respective coefficient value $\bar{c}_{i}^{l}$ at each identified position $i$ at layer $l$, and then multiply the result by an amplifying factor $q$.
By varying the value of $q$, we control the degree of resuscitation applied to the $p\%$ low-activation coefficients, thereby influencing the magnitude of the average coefficient and corresponding size of the parameter updates.

Figure 5 shows the knowledge acquisition and forgetting rates and Figure 5 presents the knowledge probe and downstream task performance after continual learning with various resuscitation configurations.
For the experiment, we fix $p$ to 50 with varying $q$ and use the OLMo checkpoint at the last step of pretraining.
Results show that when $q$ is set to 1 or greater, it generally yields better performance in both knowledge acquisition and retention compared to the original model.
In contrast, when $q$ is set to 0.5, which further reduces already inactive memory coefficients, both acquisition and retention declines suggesting that concentrating parameter updates more heavily on already active locations led to sparser updates, ultimately impairing overall performance.
These results suggest that having a narrower active memory vector (low knowledge entropy) tends to reduce the model’s capacity to acquire new knowledge and increases knowledge forgetting.

Figure: Algorithm 1 Resuscitating Low Memory Coefficients

Further experiments with fixed $q$ and varying $p$ shows that increasing $p$ to activate a larger portion of inactive parameters generally led to improved performance. Detailed result of this configuration is in Appendix B.5.
We also analyze how the result changes when using models from different stages of pretraining as the original model. We could see that the trend persists over different checkpoints from the final-stage of pretraining (554k). However, the effect of the resuscitation becomes more pronounced as the original model progresses to later stages of pretraining.
Detailed results are in Appendix B.6.

Our result indicates that resuscitating inactive memory vectors of final-stage models tends to enhance knowledge acquisition and overall performance than the final-stage model alone. However, we observe that the performance tends to be lower compared to models from the pretraining step with similar knowledge entropy, such as the mid-stage model.
This suggests that applying linear scaling to a subset of specific layers alone is insufficient to induce fundamental behavioral changes in the model. In other words, to restore a model that has lost its plasticity (Dohare et al., ) to its previous state, more fundamental and alternative approaches are required.
Further exploring methods for effectively modifying the parameters would be an interesting direction for future work.

## 5 Conclusion

In this work, we examine how large language models’ ability to broadly integrate their parametric knowledge (measured by knowledge entropy) changes throughout pretraining and how these changes affect knowledge acquisition and forgetting in a continual learning setup. Our findings reveal a strong correlation between knowledge entropy and the model’s capacity to acquire and retain knowledge. Models in the final stages of pretraining tend to exhibit narrower integration of memory vectors, leading to lower knowledge entropy, which negatively impacts both knowledge acquisition and retention. Interestingly, we could see that artificially increasing knowledge entropy by modifying the parameters of final-stage models tends to improve these capabilities.
Based on our analysis, we suggest that models from the mid-stage of pretraining offer a good balance between knowledge acquisition, retention, and overall performance, making them a good choice for further training to introduce new knowledge.

## 6 Limitation & Future Work

Due to computational constraints, our study measures knowledge acquisition and forgetting in a continual learning setup. Future work could explore whether these behaviors also occur during the pretraining phase.
We focused on OLMo 1B and 7B models, as they are the only models that publicly provide intermediate pretraining checkpoints and demonstrate strong performance (Sun & Dredze, ; Chang et al., ). Extending this investigation to other models would be a valuable direction for further research.
Our resuscitation method, which arbitrarily modifies model parameters to test our hypothesis, showed promising results in improving knowledge acquisition and retention. However, performance tended to decline when resuscitating models in their initial or mid-stages. This suggests that more refined methods for resuscitating model parameters—ones that avoid random modification and preserve language modeling capabilities—could yield better outcomes.
Additionally, while we observed that models in the mid-stage of pretraining strike a good balance for further training on tasks that involve acquiring new knowledge, defining the mid-point precisely remains an open question. In this study, we approximated the mid-point as 50% of the learning rate schedule.

#### Acknowledgments

We thank Sohee Yang, Hoyeon Chang, Seongyun Lee, Seungone Kim, Doyoung Kim, and Hanseok Oh for helpful discussions and constructive feedback.

## Appendix A Knowledge Entropy

### A.1 knowledge entropy

Figure: Figure 6: Knowledge Entropy(y-axis) measured with Dolma, C4, Pubmed corpus across different pretraining stages
Refer to caption: x8.png

##### Does the choice of dataset change the trend?

As expressed in Equation 2, knowledge entropy is dependent on the dataset $\mathcal{D}$. We define $\mathcal{D}$ as the dataset used during pretraining, as knowledge entropy reflects how the model integrates the knowledge stored in its memory vectors, learned during pretraining. However, to further explore whether the choice of dataset influences the trend of knowledge entropy, we measure it using PubMed and C4.
Figure 7 shows that the trend remains consistent regardless of the dataset used when calculating knowledge entropy.

##### Does the choice of activation function change the trend?

We also explored an alternative where we do not take the absolute value of the SwiGLU output. Instead, following the ReLU function (Agarap, ), another widely used activation function, we replaced all negative values with 0. Figure 7 shows that the trend remains consistent even under this modification.

$$ $C^{(l)}_{n,j}=\text{ReLU}(\text{gate}(\mathbf{x_{j}}))\otimes\text{up}(\mathbf {x_{j}}),$ $$

Figure: (a)
Refer to caption: x10.png

##### Layer-wise Knowledge Entropy

Figure 8 shows how knowledge entropy changes during pretraining by layer. Knowledge entropy consistently decreases in every layer, with the most significant reduction occuring in the last layer, which closely resembles the output distribution right before the token prediction. OLMo-7B also shows similar trend with 1B model.

### A.2 Entropy of Attention Layers

Inspired by previous research that emphasizes attention layer’s role of attribute extraction (Geva et al., ), we also measure attention entropy $\mathcal{H}(\theta_{\text{att}})$ similarly to Kumar & Sarawagi ()(^12^1212Kumar & Sarawagi () measures entropy using attention weights from cross-attention in an encoder-decoder architecture. However, as we employ a decoder-only model, we modify the equation to use attention weights from self-attention.). Attention weights, output of softmax normalization after key-query-value operation, can be interpreted as weight assigned to the previous tokens. As attention weight for each token position in each attention head is probability distribution(summing to 1), calculating entropy follows normal entropy formula. Then, layer-wise entropy $\mathcal{H}(\theta_{\text{att}}^{l})$ is averaged over token position and attention heads and attention entropy $\mathcal{H}(\theta_{\text{att}})$ is the sum of layer-wise entropy $\mathcal{H}(\theta_{\text{att}}^{l})$. Following the notations from Geva et al. (), attention entropy is calculated as:

$$ $\displaystyle\mathcal{H}^{n,j}(\theta_{\text{att}}^{h,l})=-\sum_{i=1}^{j} \mathbf{A}^{h,l,n}_{i,j}\cdot\log(\mathbf{A}^{h,l,n}_{i,j})\quad\text{for }i=1 ,2,\ldots,T_{n}\text{ and }j=1,2,\ldots,T_{n}$ (3) $\displaystyle\mathcal{H}(\theta_{\text{att}}^{h,l})=\frac{1}{|\mathcal{D}|} \sum_{n=1}^{|\mathcal{D}|}\left(\frac{1}{T_{n}}\sum_{j=1}^{T_{n}}\mathcal{H}^{ n,j}(\theta_{\text{att}}^{h,l})\right);\quad\mathcal{H}(\theta_{\text{att}}^{l })=\frac{1}{\text{N}}\sum_{h=1}^{\text{N}}\mathcal{H}(\theta_{\text{att}}^{h,l });\quad\mathcal{H}(\theta_{\text{att}})=\sum_{l=1}^{L}\mathcal{H}(\theta_{ \text{att}}^{l})$ $$

where $\mathbf{A}^{h,l,n}\in\mathbb{R}^{T_{n}\times T_{n}}$ represents the attention weights of the $h$-th attention head in layer $l$ for $n$-th instance, $T_{n}$ is the sequence length of the $n$-th instance in the training dataset $\mathcal{D}$, N denotes the number of attention heads, and $L$ denotes the number of layers in the model.

### A.3 Entropy of Next Token Prediction

The entropy of next token prediction (; Geng et al., ; Malinin & Gales, ) is defined as $\mathcal{H}(\theta_{\text{ntp}}^{n,j})=-\sum_{i=1}^{|\textit{V}|}p_{i}\cdot
\log(p_{i})$, where $p_{i}$ represents the probability of the $i$-th token. This is then averaged over the sequence length ($T_{n}$) and the dataset size ($|\mathcal{D}|$).

## Appendix B Knowledge acquisition and forgetting

### B.1 Datasets

##### Training Dataset for Continual Knowledge Learning

In this section, we share a brief description over the datasets we used.
For continual knowledge learning, we experiment over PubMed and C4.
The PubMed dataset consists of biomedical literature abstracts from the PubMed database, containing articles across a wide range of topics in medicine and biology.
The C4 (Colossal Clean Crawled Corpus) dataset (Raffel et al., ) is a large-scale, preprocessed collection of text scraped from the web, designed to be a clean and diverse representation of natural language.

Figure: Figure 9: Histogram of average perplexity of entities in C4 and Pubmed corpus
Refer to caption: x12.png

To compare the distribution of C4, PubMed, and Dolma, we evaluate the average perplexity of entities for C4 and PubMed, as shown in Figure 9. On the x-axis, we plot the range of an average perplexity of instances, while the y-axis represents the number of instances. We randomly sample 10,000 instances from each corpus, extract entities using GPT-4o, and calculate perplexity with the last checkpoint of OLMo. The perplexity values from the last checkpoint of OLMo indicate how likely these entities are to appear in the pretraining corpus, Dolma.
The results reveal that PubMed exhibits a broad distribution of perplexity, with a higher number of instances having high perplexity values. In contrast, C4 shows a tendency towards lower perplexity, suggesting that the distribution of entities in PubMed differs from that in Dolma, while the distribution in C4 tends to be more similar to Dolma.

##### Evaluation Dataset to measure Knowledge Acquisition

To measure knowledge acquisition of language model, we use the fictional knowledge dataset (Chang et al., ), which is designed to assess how well LLMs acquire factual knowledge during pretraining. This dataset includes 130 paragraphs
(^13^1313While Chang et al. () utilizes 120 paragraphs, comprised of 40 for paraphrase setup, 40 for duplicate, and 40 for once, we utilized 130 paragraphs, the whole original data. We scaled up the number of paragraphs with paraphrases to 70, utilizing GPT-4 following Chang et al. ())
with fictional yet realistic entities (injected knowledge) and 1,950 probes, with three different acquisition depths across five different questions for each knowledge instance, resulting in a total of 15 different probes for each sequence in the corpus. The injected knowledge is incorporated into the training corpus during continual learning, with updates occurring every 160 steps.

Following Chang et al. (), we divide the 130 corpora into two settings: paraphrase and once. In the paraphrase setting, 70 instances are each paraphrased 10 times. For every 160 steps, one paraphrased version of an instance is added to the training corpus, repeating this process 10 times(^14^1414We maintain updates every 160 steps(10 steps when batch size is 2048) because our total training duration is 1,600 steps(100 steps), and we repeat the dataset injection process 10 times.).
In the once setting, each instance is presented only once throughout the entire continual learning process. The 60 instances are divided into 10 groups, with 6 instances added every 160 steps.

##### Evaluation Dataset to measure Knowledge Forgetting

To measure forgetting rate, we evaluate over 6 downstream datasets.

- •
SciQ: multiple-choice question-answering dataset consisting of over 13,000 science exam-style questions, covering subjects like physics, chemistry, biology, and earth science
- •
Winogrande: large-scale benchmark designed to test commonsense reasoning in natural language understanding
- •
PIQA: commonsense reasoning about everyday physical interactions such as how to perform tasks involving physical actions
- •
OBQA: multiple-choice question-answering benchmark designed to assess a model’s ability to answer elementary-level science questions.
- •
HellaSwag: a large-scale benchmark for commonsense reasoning, focusing on selecting the most plausible continuation of a given narrative or scene.
- •
ARC Easy: multiple-choice science questions typically answered by students in elementary and middle school.

### B.2 Metric

In this section, we share a detailed description of how we evaluate knowledge acquisition and knowledge forgetting.

##### Knowledge Acquisition

Given a language model $\theta$, $\theta_{\text{PT}}$ represents the model extracted from a pretraining step and serves as the initial point for continual learning and $\theta_{\text{CL}}$ represents the model after it.
The acquisition metric $\mathcal{A}(\theta)$ for the model $\theta$ is defined as Equation 4.
When given a corpus set of once setting $C_{\text{once}}$, each instance in a corpus $c_{i}$ has a corresponding set of probes $P_{c_{i}}$, which contains 15 different probes.
To calculate the performance of the once setting, $\mathcal{K}_{\text{once}}(\theta)$, we compute the average log probability $\ell$($p_{i}$;$\theta$) of target span for each probe $p_{i}\in P_{c_{i}}$ across all instances in $c_{i}\in C_{\text{once}}$ and sum these averages.
The same calculation is performed for the paraphrase setting.
The total performance $\mathcal{K}(\theta)$ is calculated by the weighted average of $\mathcal{K}_{\text{once}}(\theta)$ and $\mathcal{K}_{\text{para}}(\theta)$.
Finally, the acquisition metric $\mathcal{A}(\theta)$ is defined as the improvement rate in performance $\mathcal{K}(\theta)$ from the initial model state $\theta_{\text{PT}}$ to final model state $\theta_{\text{CL}}$.

$$ $\displaystyle\mathcal{K}_{\text{once}}(\theta)=\frac{1}{|C_{\text{once}}|}\sum _{c_{i}\in C_{\text{once}}}\frac{1}{|P_{c_{i}}|}\sum_{p_{i}\in P_{c_{i}}}\ell( p_{i};\theta);\qquad\mathcal{K}_{\text{para}}(\theta)=\frac{1}{|C_{\text{para} }|}\sum_{c_{i}\in C_{\text{para}}}\frac{1}{|P_{c_{i}}|}\sum_{p_{i}\in P_{c_{i} }}\ell(p_{i};\theta)$ (4) $\displaystyle\mathcal{K}(\theta)=\frac{|C_{\text{once}}|\times\mathcal{K}_{ \text{once}}(\theta)+|C_{\text{para}}|\times\mathcal{K}_{\text{para}}(\theta)} {|C_{\text{once}}|+|C_{\text{para}}|};\qquad\mathcal{A}(\theta)=\frac{\mathcal {K}(\theta_{\text{CL}})-\mathcal{K}(\theta_{\text{PT}})}{\mathcal{K}(\theta_{ \text{PT}})}$ $$

##### Knowledge Forgetting

The forgetting metric $\mathcal{F}(\theta)$ is calculated by the average performance degradation from the initial model $\theta_{\text{PT}}$ to the final model $\theta_{\text{CL}}$ in six downstream tasks $\mathcal{T}$: SciQ (Welbl et al., ), Winograde (Sakaguchi et al., ), PIQA (Bisk et al., ), OBQA (Mihaylov et al., ), HellaSwag (Zellers et al., ), and ARC Easy (Clark et al., ).

$$ $\displaystyle\mathcal{P}(\theta)=\frac{1}{|\mathcal{T}|}\sum_{i\in|\mathcal{T} |}\mathcal{T}_{i}(\theta);\qquad\mathcal{F}(\theta)=-\frac{\mathcal{P}(\theta_ {\text{CL}})-\mathcal{P}(\theta_{\text{PT}})}{\mathcal{P}(\theta_{\text{PT}})}$ (5) $$

Figure: Figure 10: Rates of knowledge acquisition $\mathcal{A}(\theta)$ and final performance of knowledge probe $\mathcal{K}(\theta)$ according to the number of injection. Paraphrase prompt (solid line) were injected 10 times and Once (dotted line) prompt only once throughout continual training.
Refer to caption: x13.png

Figure: (a)
Refer to caption: x15.png

### B.3 Frequency of Knowledge Injections

We divide the experiment into two settings: the once setting, where knowledge is injected a single time, and the paraphrase setting, where knowledge is injected ten times using ten paraphrased paragraphs. Figure 11 shows knowledge acquisition results based on the frequency of injections. Knowledge acquisition and final performance generally follow similar trends in both settings, with models in the later stages of pretraining showing the lowest performance. However, the performance and acquisition rate of once setting lags behind that of paraphrase setting. Also, notably, for models in the final stage of pretraining, the acquisition rate in the once setting was negative. This indicates that the log probability of the injected knowledge decreased, preventing successful incorporation of the new knowledge. In other words, even the knowledge injected during continual learning is subject to forgetting throughout the continual learning process.

### B.4 Knowledge Acquisition & Forgetting

#### B.4.1 Baseline Setup

Our base experiements are reported with hyperparameter configuration most closely aligned with continual knowledge learning studies, specifically with batch size 128, learning rate 4e-4, while training single-epoch of PubMed corpus. We use adamW optimizer ($\beta$ = 0.9, 0.95, weight decay= 0.1), cosine LR scheduler with warmup=0.05, and set maximum sequence length as 1024. We randomly selected 204,800 instances from the PubMed and C4 datasets, and matched the sequence length to 1,024 tokens by concatenating instances. This resulted in a training dataset consisting of roughly 210 million tokens.

As analyzed in 4.2, final performance generally deteriorates as later-stage models were utilized as initial model. Figure 11 illustrates the model’s initial performance before continual learning, as well as its performance afterward. Models in the later stages of pretraining exhibit superior language modeling abilities before continual learning, as evidenced by the lower log probability for the newly injected knowledge (dotted line). However, after continual learning, their performance deteriorates compared to models from the earlier stages of pretraining (solid line). Similarly, the downstream task performance of the later-stage models was better initially, but after continual learning, their performance declined more than that of the earlier-stage models.

#### B.4.2 Various Settings

We observed consistently, even in settings where hyperparameters were altered, that models in the later stages of pretraining struggle to learn new knowledge and retain existing knowledge.

##### Training Dataset for Continual Knowledge Learning

To investigate how the type of new knowledge affects performance, we further conducted experiments using the C4 corpus, which has a distribution more similar to the pretraining corpus, Dolma, compared to PubMed. Figure 12 indicates that the gap in acquisition rate between later-stage models and initial-stage models is larger when the new knowledge distribution differs significantly from the pretraining corpus: models trained with PubMed (-6.4%p) exhibit a more pronounced gap compared to those trained with C4 (-3.4%p).

All models, regardless of their pretraining stage, tend to perform better when continually pretrained with PubMed compared to C4. We hypothesize that this is because PubMed’s different distribution from the pretraining corpus encourages the model to learn more new knowledge, enhancing its ability to acquire new information. However, the rate of improvement varies by model state. Later-stage models tend to show similar performance regardless of the type of new knowledge, suggesting a limit in their learning capacity. In contrast, initial-stage models exhibit a stronger ability to acquire knowledge when trained on a corpus with a different distribution, such as PubMed, demonstrating their greater adaptability in learning new and diverse information.

##### Model Size

To test universal deterioration of knowledge acquisition and retention capabilities, we also experimented with OLMo 7B model Groeneveld et al. ().
In Figure 4, the 7B model exhibits a clear trend of diminishing knowledge acquisition $\mathcal{A}(\theta)$ capabilities as pre-training progresses. This decline is accompanied by an increase in forgetting $\mathcal{F}(\theta)$, indicating that the model struggles to retain previously learned information as new data is introduced.

**Table 1: Knowledge Acquisition $\mathcal{A}(\theta)$[%] and Forgetting $\mathcal{F}(\theta)$[%] across different continual learning hyperparameters.**
|  | $\mathcal{A}(\theta)\uparrow$ | $\mathcal{F}(\theta)\downarrow$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pretraining Step of $\theta$ | 118k | 369k | 554k | 738k | 118k | 369k | 554k | 738k |
| (a) Baseline | 25.2 | 24.8 | 21.4 | 18.8 | 10.5 | 12.9 | 17.0 | 19.5 |
| (b) BS :128 $\rightarrow$ 2048 | 8.9 | 7.8 | 1.0 | -92.2 | 2.2 | 3.5 | 5.5 | 44.9 |
| (c) lr : 4e-4 $\rightarrow$ 1e-3 | 17.3 | 15.7 | 10.7 | 5.0 | 16.7 | 20.3 | 23.0 | 23.5 |
| (d) lr : 4e-4 $\rightarrow$ 1e-4 | 21.6 | 23.0 | 23.7 | 22.6 | 2.4 | 3.2 | 4.6 | 7.3 |
| (e) ep : 1 $\rightarrow$ 3 | 28.8 | 28.0 | 25.9 | 24.5 | 14.8 | 17.4 | 20.1 | 23.1 |

**Table 2: Probe Performance $\mathcal{K}(\theta)$ and average performance on six downstream task $\mathcal{P}(\theta)$ across different Continual Learning hyperparameters.**
|  | $\mathcal{K}(\theta)\uparrow$ | $\mathcal{P}(\theta)\uparrow$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pretraining Step of $\theta$ | 118k | 369k | 554k | 738k | 118k | 369k | 554k | 738k |
| (a) Baseline | -0.281 | -0.280 | -0.293 | -0.301 | 54.0 | 54.6 | 52.8 | 52.0 |
| (b) BS :128 $\rightarrow$ 2048 | -0.342 | -0.344 | -0.368 | -0.714 | 59.0 | 60.5 | 60.1 | 35.6 |
| (c) lr : 4e-4 $\rightarrow$ 1e-3 | -0.310 | -0.314 | -0.332 | -0.353 | 50.3 | 49.9 | 49.0 | 49.4 |
| (d) lr : 4e-4 $\rightarrow$ 1e-4 | -0.294 | -0.287 | -0.284 | -0.287 | 58.9 | 60.7 | 60.6 | 59.9 |
| (e) ep : 1 $\rightarrow$ 3 | -0.267 | -0.268 | -0.276 | -0.280 | 51.4 | 51.7 | 50.8 | 49.7 |

##### Batch Size

In Table 2 line (b), when the batch size is large, the influence of individual data points on the model update decreases, resulting in a lower acquisition rate, while forgetting is less pronounced. In the later stages of training, however, if sufficient learning rate warmup is not provided, the model seems to collapse.

##### Learning Rate

In Table 2 line (c), when the learning rate is increased, not only is the acquisition rate lower, but forgetting becomes more pronounced. However, when the learning rate is very small (in line (d)), the models in the late stage shows improved acquisition performance with least gap compared with initial models, but still remains inferior compared to the mid stage model.

##### Epoch

In Table 2 line (e), as the number of epochs increases, leading to more repetitions, acquisition improves with the cost of more forgetting.

Figure: (a)
Refer to caption: x17.png

### B.5 Resuscitation Experiment: Varying p 𝑝 p italic_p while Fixing q 𝑞 q italic_q

Figure 13 shows the overall performance when $q$ was fixed at 1, meaning that the lowest $p$% of coefficients were multiplied to converge toward the layer’s mean. It is shown that increasing $p$ to activate a larger portion of inactive parameters generally led to improved performance. However, interestingly, when $p$ was set too high, at 90, this resulted in a negative impact on performance, likely due to the unintended effect of reducing the parameters in already active regions.

Figure: (a)
Refer to caption: x19.png

Figure: (a)
Refer to caption: x21.png

### B.6 Resuscitation Experiment across pretraining steps

Figure 14 illustrates the overall performance when $q$ was fixed to 2 and $p$ to 50, across different pretraining stages of the original model. The effect of resuscitation method becomes more pronounced as the original model progresses to later stages of pretraining, as indicated by the transition from the dotted line to the solid line. We hypothesize that this is because late-stage models tend to rely on a smaller subset of memory sources and thus benefit from a broader scope of activation enabled by the resuscitation method. As shown in Figure 14, end performance deteriorates when the beginning model is initial (118k) and mid (369k) stage model, indicating that resuscitation may impair performance when the model’s knowledge entropy is not low enough. This trend of the resuscitation showing a more positive effect for models in later stage of pretraining can also be seen in Figure 15, which shows the result when varying $q$ while fixing $p$ as 50: performance deteriorates when running continual learning on model from 369k, while improvement of performance with larger $q$ is observed when model is from 554k.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: In L209, the negative value is often understood to remove/erase the information from the residual stream. If this flexibility is removed from the model, it could affect the model's plasticity. It would be useful to see a more flexible gating function. The entropy can still be calculated using the absolute values of the coefficients (and does not need to change the model arch and training).

What does section 3.4 do to support the use of knowledge entropy? I understand the concept of attention entropy and next token entropy but how does that connect to the choice of using knowledge entropy? In fact, if all these entropy measures decrease over time, how can we justify that knowledge entropy is the main reason for the degraded plasticity even if it shows high correlation? Section 4.3 provides interventional evidence but similar intervention should be done with, say, attention entropy to ensure this is not the source for the lowered plasticity. I.e., if making attention entropy less peaky also increase retention and mitigates forgetting, then the role of knowledge entropy is not unique as other statistics reflect the same phenomenon. Would be nice to see more interventional evidence showing that knowledge entropy is the main contributor to the plasticity issue.

Figure 5 (b): seems that downstream performance decreases as the $q$ value gets larger, which is the opposite of knowledge retention? And the knowledge probing result in the same plot doesn't show improvement over lower $q$ values. Am I missing something or does it show a different trend than Figure 5 (a)?

I'm happy to increase my scores if the above questions can be addressed.
soundness: 2
strengths: This paper investigates an important problem of knowledge acquisition and retention in LLMs. The angle is simple, clear, and easy to validate. The idea is not particularly high on the originality end but the identified "variable" (i.e., the knowledge entropy) has not been used to understand forgetting/retention as far as I know.
summary: The authors identified a phenomenon where language models use less diverse set of value vectors (i.e., the parametric memory store) to learn knowledge. The authors called the distribution "knowledge entropy" and hypothesize that the decrease in knowledge entropy hinders the models' ability to acquire and retain new knowledge. The authors found that the knowledge entropy decreases as pretraining progresses. 

The authors further test the progression of knowledge acquisition (using fictional knowledge probes) and forgetting (using common knowledge datasets) with intervention on the up-projection matrix. The author resusitate the inactive memory vectors by finding the least active $p\%$ of the memory vectors and multiply the up-projection matrix by the ratio $q$ at the corresponding index. The authors show that this reduce forgetting and increase retention of new knowledge (I have questions regarding this -- see the question section).
weaknesses: **Main Issues**

The main concern is it's unclear how causal the knowledge entropy is to the phenomenon of knowledge forgetting/retention. The paper provided mostly correlational evidence. Please see the Questions section for more concrete critique and suggestions.

**Presentation Issues**

Quite a bit of notation overloading:
1. c is used for the memory coefficient in equation 2 and used again in section 4.1 L303 to represent knowledge.
2. $m$ is used for index in equation 2 and then as multiplier in algorithm 1
L320: missing space between sentences.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: 1. Why do you think that knowledge entropy correlates with decreased ability to acquire new knowledge? 

2. Is the resuscitation method intended solely as a test for your main finding, or do you also envision a path in which this could practically be useful for increasing plasticity of models?
soundness: 4
strengths: The concept of "knowledge entropy" introduced in this paper is novel and very interesting. (1) Entropy based metrics have been used in previous works before. (2) Recent works have gradually built the view of feed-forward layers as key-value memory. This paper has a refreshed view, combining these 2 insights to give a new metric. 

The "resuscitation" method, by manipulating parameters, presents somewhat a direct test for the authors' main claim that decreased knowlege entropy gives rise to decreased ability to learn new things.
summary: This work introduces the concept of "knowledge entropy" to analyze how language models integrate their stored knowledge throughout the pretraining process. By viewing feed-forward layers as key-value memory systems, the researchers measure how broadly or narrowly models access their stored knowledge by calculating the entropy of memory coefficients. Their key finding is that as pretraining progresses, models exhibit decreasing knowledge entropy, meaning they tend to rely on a narrower set of memory vectors rather than broadly integrating knowledge from many sources. This narrowing correlates strongly with decreased ability to acquire new knowledge and retain existing knowledge during continual learning, an interesting empirical finding.
weaknesses: This work only tested on OLMo models (1B and 7B), as acknowledged by authors. The authors state their reasons for this as due to the fact that OLMo provide intermediate checkpoints. 

However the Pythia models, too, offer intermediate checkpoints. If time permits, the authors should endeavor to check their results on these models too, in order to ascertain the generality of their claims and whether it holds beyond a single model family. 

It would also be useful if the authors provide some discussion regarding mechanisms behind their main finding (knowledge entropy correlates with decreased ability to acquire new knowledge). It does not need to be theorem based, which is a difficult endeavor. Empirical findings would help, to delve at least a bit deeper into this phenomenon and why it might occur.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 2
questions: Please address the questions about bias, and also about how knowledge entropy links to intuitions about types of memories.  Finally please answer the questions about probes and knowledge acquisition.
Could you more clearly articulate how this paper builds on chang et al with respect to the part on knoweldge acquisition and forgetting?
soundness: 3
strengths: This work exposes an important feature of pretraining and contributes valuable insights to LLM pretraining.  
While the definition of knowledge entropy to this reviewer is not completely clear, it is an important feature of it that it patterns with other notions of entropy in models like attention entropy or next token prediction entropy.   The work on knowledge acquisition and forgetting is potentially very important for how we view model training.


The empirical results are solid and convincing to this reviewer.
summary: This paper investigates how knowledge acquisition and forgetting evolves through an LLM's pretraining. Building on the work of Chang et al 2024 and Geva et al 2021 and 2022, the authors introduce a concept of knowledge entropy and show that low entropy, which corresponds to poor knowledge acquisition and high forgetting rate, occurs in late stages of pretraining while high entropy occurs for initial stages of pretraining.
weaknesses: This paper has the potential to make substantial contributions to the NLP and ML communities.  But I believe it still could benefit from further work.

The talk of broad and particular memory vectors in the introduction is a bit unclear.

In calculating entropy, the authors assume the feed forward network has the form
FFN(x) = f(x · K^T ) · V
but this leaves out important bias terms that might perhaps affect the calculations.  Can the authors say more about this?


There are some important connections with Chang et al 2024 that need to be brought out more.  As it stands, it seems as though much of the heavy lifting is done in Chang et al, and this paper provides an incremental improvement to that paper.  There needs to be a discussion of how this paper goes beyond the previous one.  

Also it is important to not oversell the information contained in the so called memory vectors.  In particular, given the experiments with human interpreters given the strings that affect these memory vectors the most,  what we get are suggestive interpretations of what's stored, that is suggestive to those particular human interpreters.  As far as I know Chang et al shows mostly  it's not clear that the model can make use of them in the way they are interpreted.

While it's pretty clear intuitively what attention entropy or next token prediction entropy are, it's less clear at least to this reviewer what the intuition is behind defining entropy in terms of the coefficients of the outputs of a given FFN layer in the model.   How does the knowledge entropy so defined link up with the suggestion that "models in the later stage of pretraining tends to engage with a narrower
range of memories, relying more heavily on specific memory vectors rather than accessing and integrating
knowledge from a broader range of memories"?

Another element that needs a bit more work is the discussion of knowledge acquisition and forgetting.  This relies on work from Chang et al. but to be more self-contained the authors could explain in a bit more detail what the probes are doing (and why, for instance, 15 different probes?), or rather summarize Chang et al's contributions more clearly with respect to the contributions of the present work.

That part as it stands leaves this reader wanting more information.  For instance, what is the target span for the probes?  Also is \theta_{pt} unique? Is there only one such check point and what about \theta_{cl}?  Is \theta_{cl} when the continual learning stops, if it does, or what?  Is  \theta_{cl} then the model after pre training?   There is more material in the appendix on this but it would help the reader's understanding of this important concept of knowledge acquisition and knowledge loss  to move some of that discussion into the body of the paper.

Some minor stylistic corrections that will render the paper more readable:

Works ---> work   (`work' is still a mass known in English, as far as I know)
coefficient of j-th token  ---> coefficient of the  j-th token

into probability distribution --> into a probability distribution

We also experiment that the trend persists ---> We also show experimentally that the trend persists

we inject FICTIONAL KNOWLEDGE dataset  ---> we inject elements of the FICTIONAL KNOWLEDGE dataset??

of target span  ---> of the target span

Equation and detailed explanation is
presented  ---> please fix the English

Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong Chang, and Minjoon
Seo. How do large language models acquire factual knowledge during pretraining? arXiv preprint
arXiv:2406.11813, 2024.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Given that consistent behavior is observed across all layers, do you think we can use only a subset of layers to determine the model's proneness to learning new knowledge? For example, only focusing on layers that mainly encode semantic knowledge as shown in the literature. 
- Related to the question above, is the same trend observed across different forms of knowledge and is there a difference between various knowledge forms (syntactic knowledge versus semantic knowledge for example)? 
- Given the minor negative effects of resuscitation of inactive vectors, do you think this is a viable method for adaptation in continual learning frameworks in generals to minimize forgetting and increasing knowledge acquisition? More specifically, what negative effects can this artificial resuscitation have?
- In Figure 5(b), what is the reason behind the decrease in the downstream task performance while a drastic decrease in forgetting is also observed? Shouldn't a low forgetting rate lead to better general-purpose performance?
soundness: 3
strengths: - The proposed framework of categorizing knowledge retention in terms of the model's knowledge entropy is interesting albeit somewhat expected as it is generally understood that moving away from established weight matrices becomes more difficult as the training progresses.
- The concept of entropy in this case is both well-defined and intuitive, essentially showing how variant the coefficients are with respect to each input token given an instance.
- The experimental results confirm the theoretical hypothesis, strongly indicating that knowledge entropy is associated with both learning and forgetting.
summary: The paper introduces knowledge entropy as a novel concept for the quantification of model representation across Large Language Models. The authors show that as the training of a large model progresses, the knowledge tends to concentrate in a smaller set of "memory vectors", making the process of knowledge acquisition difficult (decrease in plasticity), and the model more prone to forgetting prior knowledge as memory vectors are forced to be constantly rewritten. Experimental results show that it is indeed the case that models in the later stages of their training are more resilient to learning new knowledge, and tend to easily forget the knowledge that they already have. In addition, by resuscitating the inactive vectors, the authors force a later-stage model to ameliorate both the aforementioned concerns in the continual learning process.

Overall, I think this is a good paper with interesting results that can have further applications for the community. Some concerns and questions still remain as outlined below, but I am willing to increase my score once they are addressed.
weaknesses: - The usage of the term "entropy" in this case can be confusing in my opinion. Usually, this term implies some form of "surprise" with respect to information. However, in this case, it seems to refer to the idea of source broadness. 
- I think this is a paper that could benefit greatly from further evaluation with respect to knowledge entropy. Although it is expected that the same behavior is observed across models, I would like to know if different architectures affect the observed behavior and if different training time design choices can regulate the knowledge entropy on a case-by-case basis.
- Further discussion is required regarding the correspondence of knowledge entropy and forgetting. Rather than being a direct consequence of knowledge entropy, can it be the case that the reason behind sharper declines in knowledge retention is due to the model's richer knowledge representation between concepts at later stages of the training and thus being more prone to knowledge loss when small changes are made? More specifically, forgetting at a higher rate at later stages can be expected simply due to having more knowledge encoded in comparison to having very little knowledge in earlier stages. 
- Minor grammatical errors that can be fixed in the camera-ready version.
- I believe that Algorithm 1 can be rewritten to better relay the information contained. Namely, by giving a better explanation of each operation (especially with respect to $t$).

Meta-review: This paper introduces the concept of "knowledge entropy" to analyze how language models store and access knowledge and present empirical evidence showing that knowledge entropy decreases as models are trained, correlating with a reduced ability to learn new information and an increased tendency to forget existing knowledge.

This is a novel approach that provides useful insights to an relevant problem in LM pre-training. The experiments are novel / creative and well supported. Reviewers agree that this is a valuable addition to the literature.

Weaknesses like notation overload and relation to previous work were addressed in rebuttal.

Decision: Accept (Oral)