Gradual Learning: Optimizing Fine-Tuning with Partially Mastered Knowledge in Large Language Models

## Abstract

Abstract During the pretraining phase, large language models (LLMs) acquire vast amounts of knowledge from extensive text corpora. Nevertheless, in later stages such as fine-tuning and inference, the model may encounter knowledge not covered in the initial training, which can lead to hallucinations and degraded performance. This issue has a profound impact on the model’s capabilities, as it will inevitably face out-of-scope knowledge after pretraining. Furthermore, fine-tuning is often required to adapt LLMs to domain-specific tasks, necessitating the acquisition of new knowledge. However, this phenomenon limits the model’s ability to learn and integrate new information during fine-tuning. The effectiveness of fine-tuning largely depends on the type of knowledge involved. Existing research suggests that fine-tuning the model on partially mastered knowledge—for instance, question-answer pairs where the model has a chance of providing correct responses under non-greedy decoding—can enable the model to acquire new knowledge while mitigating the forgetting of previously learned information. Notably, this approach can still lead to the forgetting of fully mastered knowledge, constraining the fine-tuning dataset to a narrower range and limiting the model’s overall potential for improvement. Given the model’s intrinsic reasoning abilities and the interconnectedness of different knowledge areas, it is likely that as the model’s capacity to utilize existing knowledge improves during fine-tuning, previously unmastered knowledge may become more understandable. To explore this hypothesis, we conducted experiments and, based on the results, proposed a two-stage fine-tuning strategy. This approach not only improves the model’s overall test accuracy and knowledge retention but also preserves its accuracy on previously mastered content. When fine-tuning on the WikiQA dataset, our method increases the amount of knowledge acquired by the model in this stage by 24% .

## 1 Introduction

In recent years, large language models (LLMs) have advanced rapidly, demonstrating exceptional performance across a variety of tasks Devlin et al. (); Achiam et al. (); Dubey et al. (); Yang et al. (). These models are pretrained on vast amounts of text data, allowing them to encode substantial knowledge within their parameters Petroni et al. (); Roberts et al. (); AlKhamissi et al. (). Following pretraining, LLMs undergo supervised fine-tuning to improve their ability to follow instructions Zhao et al. (); Zhang et al. () and tackle domain-specific tasks He et al. (); Cui et al. (); Li et al. ().

However, unlike the pretraining phase, LLMs face challenges when encountering new knowledge during other stages, including fine-tuning and inference. When fine-tuning data includes knowledge unrelated to the data seen during the pretraining phase, LLMs struggle to learn this new information, making them prone to overfitting and generating hallucinations. Gekhman et al. (); Gudibande et al. (). Additionally, even when external knowledge is integrated into prompts during inference, such as through KG-enhanced LLMs Pan et al. () or Retrieval-Augmented Generation (RAG)Zhao et al. (), these models often struggle to manage unfamiliar knowledge effectivelyKang et al. (); Lee et al. (); Yin et al. (). These characteristics impose the following constraints on the design of LLMs:

- •
Constraint 1: During the fine-tuning stage, the knowledge in the data should be associated with the knowledge from the pretraining phase to mitigate overfitting and reduce hallucinations.
- •
Constraint 2: During the inference stage, if external knowledge is to be provided in prompts, it must be related to the knowledge the model has already learned.

In this data-driven era Bai et al. (), these constraints fundamentally limit the data utilization during fine-tuning and inference, thereby significantly restricting the capabilities of LLMs. To mitigate the impact of these constraints on model performance, we propose a novel two-stage fine-tuning method based on the interconnections between knowledge and the model’s reasoning ability. This approach aims to broaden the pool of data suitable for training during the fine-tuning stage.

In comparison to smaller models, LLMs exhibit surprising emergent abilities, such as reasoning Wei et al. (). For instance, we can harness the knowledge reasoning capability of LLMs by utilizing KG to enhance the inputs of these models, thereby improving their performance Pan et al. (). Additionally, the knowledge within these models is not isolated but rather interconnected Ji et al. (). Based on these observations, we hypothesize that if an LLM acquires knowledge that was not fully mastered—knowledge the model is likely to answer correctly without fine-tuning, during self-supervised fine-tuning, it could potentially leverage its existing knowledge to comprehend additional concepts that were neither previously mastered nor present in the training dataset. To test this hypothesis, we conducted experiments using mainstream LLMs on a closed-book question-answering task and validated our conjecture. Building on these findings, we designed a two-stage fine-tuning framework: in the first stage, the model is fine-tuned using knowledge that it partially understands; in the second stage, we monitor changes in the types of knowledge within the training set and use the data with improved mastery as augmented training data for a second stage of fine-tuning. The experimental results illustrate that our approach substantially improves the model’s accuracy on test data and its mastery of knowledge within the training dataset.

Our contributions can be summarized as follows:

- •
We propose that fine-tuning can enhance a LLM’s mastery of knowledge that it previously did not fully comprehend and that was not present in the original training dataset. We conducted experiments to validate this hypothesis.
- •
Building on this, we utilized the data with improved mastery to augment the training dataset and performed a subsequent round of fine-tuning on the LLM, which can significantly enhance knowledge acquisition and lead to increased accuracy on the test set. This method broadens the range of data available for fine-tuning LLMs, offering a new perspective from a data-driven standpoint to enhance the performance.

## 2 Background and Related Work

### 2.1 Impact of New Knowledge on Hallucination

LLMs can acquire substantial knowledge during the pretraining phase, enabling them to function as knowledge bases for downstream tasks AlKhamissi et al. (). However, these models are prone to a phenomenon known as hallucination, where the generated content, while syntactically correct and seemingly coherent, conflicts with contextual information or real-world knowledge. The causes of hallucinations are varied, including the use of low-quality datasets and suboptimal data utilization Huang et al. ().

It is widely argued that LLMs primarily acquire knowledge during the pretraining phase, while processes such as supervised fine-tuning and in-context learning are more focused on the extraction and application of existing knowledge, with limited capacity for learning new knowledge. Gudibande et al. Gudibande et al. () found that during supervised fine-tuning, LLMs predominantly learn the stylistic aspects of the fine-tuning dataset rather than factual knowledge. This can lead the model to confidently generate content that is not factually accurate. Research by Kang et al. Kang et al. () suggests that unfamiliar queries can trigger hallucinations in LLMs and that fine-tuning the model on data not covered during pretraining can influence the type of hallucinations produced. Experiments by Yin et al. Yin et al. () and Lee et al. Lee et al. () indicate that using unfamiliar knowledge as a template in in-context learning can degrade the performance of LLMs in such tasks. Gekhman et al. Gekhman et al. () categorized knowledge into four distinct types based on the model’s probability of correctly answering corresponding questions under different decoding strategies. Their experiments unveiled that fine-tuning the model on knowledge that the model has not yet mastered-knowledge that the model cannot correctly answer under a greedy decoding strategy—results in overfitting and notably diminishes the model’s accuracy in responding to questions related to mastered knowledge—knowledge that the model can correctly answer under a greedy decoding strategy. Their research indicates that fine-tuning on knowledge that the model is likely to answer correctly enhances test set accuracy, mitigates overfitting, and sustains performance on previously acquired knowledge, as opposed to fine-tuning on all data or a specific type of data. Our methods followed the knowledge classification proposed by Gekhman et al.

Overall, encountering new knowledge after the pretraining phase poses a significant challenge for LLMs, making it difficult both to learn this knowledge and to utilize it effectively during fine-tuning and inference.

### 2.2 Quantifying LLM Knowledge Mastery

Assessing the degree to which LLMs have mastered specific knowledge is crucial for both understanding the internal mechanisms of these models and constructing better ones. In scenarios where the ground truth is known, such as in closed-book question-answering tasks, the model’s knowledge can be evaluated by generating multiple responses and measuring accuracy Kadavath et al. (); Gekhman et al. (). However, in cases where ground truth is unavailable, such as in real-world applications of LLMs, alternative evaluation methods are required. The confidence score is often used to assess a model’s certainty in its specific outputs (such as words, phrases, or sentences) and can serve as an indicator of the model’s knowledge mastery. Ideally, a confident score should be well-calibrated, meaning it can accurately reflect the probability that an answer is correct in the real world. Nevertheless, since LLMs are often trained with Reinforcement Learning from Human Feedback (RLHF) Ouyang et al. (), they may confidently produce incorrect information, leading to conditional probabilities that are not well-calibrated. Researchers have proposed various methods to address this issue, such as calculating the degree of agreement among different generated outputs Lyu et al. (), training models to compute confidence scores based on activation states Azaria and Mitchell (), or directly querying the LLM about its confidence in its knowledge Tian et al. (). Notably, Kadavath et al. Kadavath et al. () pointed out that the accuracy of repeated responses is a well-calibrated metric for assessing the model’s knowledge, which is also the method we adopted to gauge the extent of knowledge mastery.

### 2.3 Continue Fine-tuning

Our approach involves the issue of continual fine-tuning, where after fine-tuning the model, further fine-tuning is conducted using new data. Continual learning is crucial for models that need to adapt over time, learning from a continuous stream of data without forgetting previously acquired knowledge. However, one of the most significant challenges in this area is the phenomenon known as catastrophic forgetting. This occurs when a model, after being exposed to new data, experiences a significant degradation in performance on tasks it had previously learned and mastered. As the model adjusts its parameters to accommodate new information, it inadvertently overwrites the information and skills it had previously acquired, leading to a loss of accuracy and effectiveness in those earlier tasks. To mitigate this issue, researchers have proposed various methods, broadly categorized into four types Zheng et al. (): Replay-Based Shin et al. (); Ren et al. (), Regularization-Based Mi et al. (), Gradient-Based Lee et al. (), and Architecture-Based Geng et al. () approaches. In our experiment, we employed basic experience replay, appropriately lowered the initial learning rate, and set weight decay parameters to prevent overfitting to the new data.

## 3 Methods

Our experiments are divided into two parts. The first part examines the changes in knowledge types after fine-tuning. After validating our hypotheses in this initial stage, the second part proposes a two-stage fine-tuning strategy based on these findings. Detailed descriptions of our methods are provided in Sections 3.1 and 3.2, respectively. The overall process is illustrated in Figure 1.

Figure: Figure 1: Figure (a) illustrates the method of knowledge classification. For each question-answer pair, multiple prompts were constructed, and the model was queried several times using both greedy and non-greedy decoding methods. Knowledge was then classified based on the accuracy of the responses and the corresponding decoding strategies. Figure (b) depicts the two-stage training process: the first stage involves fine-tuning with knowledge classified as ‘Maybe Known’, followed by a re-detection of knowledge categories. Based on the re-detection, the training data is augmented, and the model undergoes a second stage of fine-tuning, ultimately resulting in a large language model with a broader mastery of knowledge.
Refer to caption: x1.png

### 3.1 Detect Knowledge Type Change

Knowledge is interconnected, and it is possible to infer new knowledge from existing information. We propose that during the fine-tuning process, the model may acquire knowledge that it originally did not possess and that is not present in the training dataset for the following reasons:

- •
Some knowledge not included in the training dataset and initially unknown to the model can be inferred from the knowledge that is present within the training dataset.
- •
Even if certain knowledge in the training dataset cannot directly infer other knowledge, it can be combined with information acquired by the model during the pretraining phase to infer knowledge that is absent from the training dataset and was originally unknown to the model.

To validate our hypothesis, we first classified the knowledge into four categories—‘Highly Known’, ‘Maybe Known’, ‘Weakly Known’, and ‘Unknown’—based on the work of Gekhman et al. Gekhman et al. (), with the specific criteria outlined in Table 1. We then fine-tuned the model using data classified as ‘Maybe Known’. At the end of each epoch, we evaluated the model’s accuracy on the test set. Once the accuracy reached its maximum, we re-evaluated the knowledge types.

**Table 1: Knowledge classification according to Gekhman et al. $(q,a)$ represents a question-answer pair, and $T$ denotes the temperature parameter during decoding. For greedy decoding, $P_{correct}$ represents the probability of the model correctly answering a question $q$. to estimate the probability of correctly answering questions, ten generations were performed. For each generation, prompts were randomly constructed, with each prompt containing four examples. For non-greedy decoding, the temperature was set to 0.5, with a sampling size of 16 and a top-k value of 40. Ten generations were conducted under these conditions as well.**
| Type | Defination | Explanation |
| --- | --- | --- |
| Highly Known | $P_{correct}(q,a;T=0)=1$ | can be correctly answered using a greedy decoding strategy |
| Maybe Known | $P_{correct}(q,a;T=0)\in(0,1)$ | might be correctly answered using a greedy decoding strategy |
| Weakly Known | $P_{correct}(q,a;T=0)=0\land\newline<br>P_{correct}(q,a;T>0)>0$ | might be correctly answered by a non-greedy decoding strategy, but not by a greedy one |
| Unknown | $P_{correct}(q,a;T=0)=0\land\newline<br>P_{correct}(q,a;T>0)=0$ | cannot be correctly answered even with a non-greedy decoding strategy |

### 3.2 Two-stage Fine-tuning

Based on the experimental results presented in Section 3.1, which will be detailed in Section 4.1, we observed that following fine-tuning, a significant portion of knowledge showed improved mastery, while some knowledge exhibited a decline in mastery. Building on these observations, we proposed a two-stage fine-tuning strategy.
In the second stage of fine-tuning, we used data categorized as ‘Maybe Known’ at the end of the first stage, which includes data originally classified as ‘Weakly Known’ and ‘Unknown’ that were excluded from the initial training dataset. We continued fine-tuning with this dataset while implementing a replay strategy with a 0.2 replay ratio for data categorized as ‘Highly Known’ after fine-tuning. Specifically, at the start of each epoch, we sampled a portion of the ‘Highly Known’ data according to the replay ratio, mixed it with the rest of the training data, and shuffled it for training. This strategy not only expands the training data but also helps prevent the forgetting of knowledge that the model has already mastered.

## 4 Experiments and Analysis

Sections 4.1 and 4.2 correspond to the experiments and their analyses discussed in Sections 3.1 and 3.2, respectively.

### 4.1 Detect Knowledge Type Change

#### 4.1.1 Experimental Setup

At this stage, we broadly followed the experimental methodology of Gekhman et al. We conducted a closed-book question-answering task using the WikiQA dataset Yang et al. (). The only adjustment we made to the dataset was to train on question-answer pairs with a single answer. During testing, for questions with multiple answers, we recorded a response as correct if the model accurately generated the first answer. For each data point in the training set, we randomly selected four other questions of the same type to create a prompt template and performed multiple generations to assess accuracy. The generation parameters were consistent with those outlined in Table 1. To account for cases where the model generated extra content, we considered an answer correct if the correct response appeared as a substring in the model’s output. We used VLLM Kwon et al. () to accelerate inference when determining the types of knowledge.

During fine-tuning, we employed LoRA Hu et al. () with a rank setting of 64. We used the AdamW optimizer, a cosine scheduler, and set the initial learning rate to 3e-4. All training was conducted on four A100 GPUs, with inference performed on a single A100 GPU, each with 80GB of memory. The batch size was set to 32.

We primarily used the Qwen2 model Yang et al. (), a leading open-source model, for our experiments. When the model name is not explicitly mentioned, the model being used is Qwen2-7B. To further validate our conclusions, we also conducted trials using the LLaMA3 model Dubey et al. ().

#### 4.1.2 Experiment Results and Analysis

The number of each type of knowledge in the training dataset before fine-tuning is shown in Table 2.After fine-tuning for several epochs, we evaluated the model’s accuracy on the test set at the end of each epoch. Upon accuracy convergence, we examined the changes in the attributes of each knowledge point. Table 3 presents the experimental results using Qwen2 and LLaMA3.

**Table 2: The number of knowledge points for each type before fine-tuning.**
| Model | Type | Num | Model | Type | Num |
| --- | --- | --- | --- | --- | --- |
| Qwen2-7B | HighlyKnown | 34426 | LLaMA3-8B | HighlyKnown | 34739 |
| MaybeKnown | 36897 | MaybeKnown | 42400 |  |  |
| WeaklyKnown | 25986 | WeaklyKnown | 24701 |  |  |
| Unknown | 71855 | Unknown | 67324 |  |  |

**Table 3: A statistical analysis of the changes in knowledge types after the first stage of fine-tuning. Since distinguishing between ‘Weakly Known’ and ‘Unknown’ is costly and less relevant to our approach (as we only need to differentiate between ‘Highly Known’, ‘Maybe Known’, or neither when selecting training data), these two categories were grouped together. We have highlighted in bold the parts where the mastery level transitions from a lower level to "Maybe Known."**
| Model | Origin Type | New Type | Num |
| --- | --- | --- | --- |
| Qwen2-7B | HighlyKnown | HighlyKnown | 27282 |
| MaybeKnown | 3952 |  |  |
| WeaklyKnown & Unknown | 3192 |  |  |
| MaybeKnown | HighlyKnown | 19059 |  |
| MaybeKnown | 9892 |  |  |
| WeaklyKnown & Unknown | 7946 |  |  |
| WeaklyKnown | HighlyKnown | 3091 |  |
| MaybeKnown | 4889 |  |  |
| WeaklyKnown & Unknown | 18006 |  |  |
| Unknown | HighlyKnown | 527 |  |
| MaybeKnown | 1932 |  |  |
| WeaklyKnown & Unknown | 69396 |  |  |
| LLaMA3-8B | HighlyKnown | HighlyKnown | 29930 |
| MaybeKnown | 4428 |  |  |
| WeaklyKnown & Unknown | 381 |  |  |
| MaybeKnown | HighlyKnown | 19635 |  |
| MaybeKnown | 19976 |  |  |
| WeaklyKnown & Unknown | 2789 |  |  |
| WeaklyKnown | HighlyKnown | 1991 |  |
| MaybeKnown | 8607 |  |  |
| WeaklyKnown & Unknown | 14103 |  |  |
| Unknown | HighlyKnown | 396 |  |
| MaybeKnown | 5165 |  |  |
| WealyKnown & Unknown | 61763 |  |  |

**Table 4: The differences in results when testing knowledge types twice using the same model.**
| Origin Type | New Type | Num |
| --- | --- | --- |
| HighlyKnown | HighlyKnown | 31987 |
| MaybeKnown | 3190 |  |
| WeaklyKnown &Unknown | 0 |  |
| MaybeKnown | HighlyKnown | 3285 |
| MaybeKnown | 30006 |  |
| WeaklyKnown & Unknown | 5236 |  |
| WeaklyKnown & Unknown | HighlyKnown | 0 |
| MaybeKnown | 5285 |  |
| WeaklyKnown & Unknown | 97571 |  |

Analysis of the results presented in the tables reveals the following:

- •
After several epochs of fine-tuning, a significant portion of knowledge points originally classified as ‘Weakly Known’ or ‘Unknown’ were reclassified as ‘Maybe Known’, with some even transitioning to ‘Highly Known’. It is important to note that the model was trained exclusively on data labeled as ‘Maybe Known’.
- •
Some data points initially classified as ‘Highly Known’ were reclassified as ‘Maybe Known’, or even to lower categories. This indicates the occurrence of forgetting during the fine-tuning process.

Since we use the frequency of correctly answered questions as a proxy for the model’s knowledge mastery, and given that prompt construction during testing is random, with the sampling process being stochastic when setting the temperature parameter, we may observe changes in the classification of some knowledge points even when retesting the same model. To determine how much of these attribute changes are due to training, we retested the knowledge categories using the model without fine-tuning, as shown in Table 4. Although individual knowledge points may change categories between different tests, the number of knowledge points whose categories change remains fairly stable due to the concentration inequality. Therefore, the results of a single retest can indicate the extent to which random factors contribute to changes in knowledge categories. Analysis of the table suggests that a significant portion of the category changes can be attributed to fine-tuning.

To investigate the possible reasons behind these changes, we constructed a graph. For question-answer pairs initially classified as ‘Maybe Known’, as well as those initially classified as ‘Weakly Known’ but reclassified as ‘Maybe Known’ after fine-tuning, we represented the entities in both the answers and the questions as nodes. For example, in the question-answer pair “Question: Who performed Rodney Crowell - Greatest Hits? Answer: Rodney Crowell", we extracted ‘Rodney Crowell - Greatest Hits’ and ‘Rodney Crowell’ using regular expressions and used them as nodes. For each question-answer pair, we connected the entities in the question and answer with an undirected edge. We categorize the nodes into the following three types:

- •
Initial: Nodes from question-answer pairs initially classified as ’Maybe Known’.
- •
Reclassified: Nodes whose classification changed from ’Weakly Known’ to ’Maybe Known’.
- •
Linked Reclassified: Nodes reclassified from ’Weakly Known’ to ’Maybe Known’ and connected to ’Initial’ nodes.

The Table 5 shows that after several epochs of training, most nodes that transitioned from ‘Weakly Known’ to ‘Maybe Known’ are closely connected to nodes originally classified as ‘Maybe Known’, which aligns with our hypothesis.

**Table 5: The connections between knowledge initially classified as ‘Maybe Known’ and the knowledge that transitioned from ‘Weakly Known’ to ‘Maybe Known’ after the first stage of fine-tuning.**
| Label Type | Initial | Reclassified | Linked Reclassified |
| --- | --- | --- | --- |
| Count | 43,005 | 5,121 | 3,968 |

### 4.2 Two-Stage Fine-tuning

#### 4.2.1 Experimental Setup

During continual fine-tuning, we reduced the initial learning rate to 15e-5 and set the weight decay to 0.01. We set the replay ratio to 0.2 and continued using LoRA with a rank of 64. To minimize the influence of random factors when evaluating the model’s accuracy, we generated a fixed prompt test set at the beginning of the experiment using random seed 42.

#### 4.2.2 Experiments and Analysis

We first validated the limitations of one-stage fine-tuning, specifically when using only data initially classified as ‘MaybeKnown’, as also recommended by Gekhman et al., which outperforms training on all data or other types of data. Our results, shown in Table 7, indicate that increasing the number of fine-tuning epochs does not improve model performance. It can be observed that after 10 epochs (in fact, the highest test accuracy is achieved by the end of the eighth epoch), increasing the number of training epochs does not enhance accuracy when using an early stopping strategy. Moreover, without early stopping, extending the training epochs leads to overfitting on the training set, resulting in a decline in test accuracy.

**Table 6: The peak accuracy and final accuracy at the end of training across different epochs during fine-tuning on data classified as ‘Maybe Known’.**
| Training Epoch | Max Acc | Final Acc |
| --- | --- | --- |
| 10 | 32.98 | 32.74 |
| 15 | 32.99 | 31.21 |
| 20 | 33.00 | 30.73 |
| 25 | 32.99 | 30.50 |

We then conducted the two-stage fine-tuning experiment, collecting the maximum accuracy achieved after the second stage (typically occurring at the end of the first to third epochs), and compared it with the results from the one-stage fine-tuning. The results are presented in Table 7. The experimental results demonstrate that our method significantly improves the model’s test accuracy.

Since the data used for two-stage fine-tuning includes knowledge classified as ‘Weakly Known’ after the first stage as well as knowledge classified as ‘Highly Known’, we aimed to determine whether the improvement in model accuracy after the two-stage fine-tuning is due to the acquisition of new knowledge or the reduction of forgetting through data replay of previously mastered knowledge. To investigate this, we designed various two-stage fine-tuning strategies and conducted experiments. The different strategies are outlined below, and the experimental results are presented in Table 8.

- •
Strategy 1: Fine-tuning with data that remained classified as ‘Maybe Known’ after fine-tuning and was initially categorized as ‘Highly Known’, ‘Maybe Known’, or ‘Weakly Known’.
- •
Strategy 2: Fine-tuning with data that remained classified as ‘Maybe Known’ after fine-tuning and was initially categorized as ‘Highly Known’ or ‘Maybe Known’.
- •
Strategy 3: Fine-tuning with data that remained classified as ‘Maybe Known’ after fine-tuning and was initially categorized as ‘Maybe Known’ or ‘Weakly Known’.
- •
Strategy 4: Fine-tuning with data that remained classified as ‘Maybe Known’ after fine-tuning.
- •
Strategy 5: Fine-tuning with data that remained classified as ‘Maybe Known’ after fine-tuning, while replaying 20% of data classified as ‘Highly Known’ after fine-tuning. Strategy 5 is the one used in the previous experiments.

**Table 8: The maximum accuracy achieved after second-stage fine-tuning using datasets constructed with different strategies.**
| Strategy | One-Stage | Strategy 1 | Strategy 2 | Strategy 3 | Strategy 4 | Strategy 5 |
| --- | --- | --- | --- | --- | --- | --- |
| Max Acc | 32.98 | 33.62 | 33.24 | 33.29 | 33.62 | 33.80 |

To explore the impact of two-stage fine-tuning on the model’s accuracy concerning knowledge it originally mastered, we classified the data in the test set using the un-tuned model and then compared the model’s test accuracy on different types of knowledge after applying different strategies for two-stage fine-tuning. The results are shown in Table 9.

**Table 9: The model’s accuracy on test data across different categories, which were determined using the un-finetuned model. It can be observed that all strategies, except for Strategy 2, improve the model’s accuracy across all categories of test data during the second stage of fine-tuning. Strategy 2 did not utilize data that was initially classified as ‘Weakly Known’ but showed improved mastery.**
| Stage | HighlyKnown | MaybeKnown | WeaklyKnown | Unknown |
| --- | --- | --- | --- | --- |
| One-Stage | 85.03 | 56.90 | 18.38 | 1.62 |
| Strategy 1 | 86.76 | 56.94 | 19.32 | 1.95 |
| Strategy 2 | 86.96 | 56.77 | 17.86 | 1.59 |
| Strategy 3 | 85.67 | 57.06 | 18.88 | 1.77 |
| Strategy 4 | 86.59 | 56.35 | 19.90 | 2.13 |
| Strategy 5 | 86.94 | 57.00 | 19.64 | 2.15 |

Based on the results presented in the Table 8 and Table 9, we can conclude the following:

- •
Whether by using the forgotten data after fine-tuning as additional data for continued fine-tuning, employing newly improved data as supplementary data, or combining both approaches, it is possible to effectively enhance the model’s performance. The improvement in model performance after two-stage fine-tuning results from both the acquisition of new knowledge and the mitigation of forgetting existing knowledge during the fine-tuning process.
- •
Combining both approaches—augmenting the training data with knowledge that shows improved mastery after the first stage of fine-tuning, while simultaneously replaying fully mastered data to mitigate forgetting—proves to be the most effective strategy.
- •
Regardless of whether data initially classified as ‘HighlyKnown’ is used during the second stage of fine-tuning, the model’s accuracy on test data initially classified as ‘HighlyKnown’ is not diminished—in fact, it is improved.

While testing the model’s accuracy on the test set can effectively reflect the model’s capabilities, it does not entirely align with our objective. Our aim is to expand the amount of data that LLMs can utilize during supervised fine-tuning, alleviate the negative impact of the model’s weaker learning ability during the fine-tuning stage, and thereby enhance the potential of LLMs from a data perspective. Hence, in addition to the accuracy on the test set, we also need to understand the model’s mastery of knowledge in the training set. Therefore, we also tracked the number of knowledge points of each type in the training set at different stages, as presented in Table 10. The findings suggest that compared to one-stage fine-tuning, two-stage fine-tuning can:

- •
significantly increase the number of knowledge classified as ’Highly Known’ (approximately 7.5%.If solely considering incremental enhancements, it reaches approximately 24%.), representing question type that the model is highly likely to answer correctly.This indicates that the method significantly enhances the knowledge mastered by the model.
- •
reduces the proportion of data classified as ‘Weakly Known’ and ‘Unknown’. This implies an expansion of the range of data that can be used for fine-tuning.

**Table 10: The number of different types of knowledge at various stages.From the table, it can be observed that after two-stage fine-tuning, the classification of ‘Highly Known’ has significantly increased, while the other types have decreased.**
| Type | Origin | One-Stage | Two-Stage |
| --- | --- | --- | --- |
| HighlyKnown | 34426 | 49959 | 53691 |
| MaybeKnown | 36897 | 20665 | 18288 |
| WeaklyKnown & Unknown | 97841 | 98540 | 97185 |

#### 4.2.3 Multiple-Rounds Fine-Tuning Attempts

Building on the most effective Strategy 5, we attempted multiple rounds of fine-tuning. After completing the two-stage fine-tuning process, we again assessed the changes in knowledge categories and iteratively applied the fine-tuning method described earlier, aiming to maximize the model’s performance. However, the results indicated that multiple rounds of fine-tuning did not lead to further improvements in model performance. The results are shown in Table 11.

**Table 11: Accuracy results for one-stage, two-stage, and multi-round fine-tuning.**
|  | One-Stage | Two-Stage | Multi-Rounds |
| --- | --- | --- | --- |
| Accuracy | 32.98 | 33.80 | 33.79 |

We hypothesize that this phenomenon may be due to the fact that, after the first round of fine-tuning, the model has already improved its mastery of knowledge as much as possible given its current capabilities and the content of the dataset. After the second round of fine-tuning, there was little further improvement in knowledge mastery. The lack of new data for continued fine-tuning led to a scarcity of fresh information, and further training on the existing data resulted in overfitting. We analyzed the changes in knowledge types in Table 12. As seen in Table 12, compared to the first stage, the number of knowledge types that changed before and after the second stage of fine-tuning significantly decreased, supporting our hypothesis. This rapid convergence also highlights the efficiency of our method.

**Table 12: The changes in knowledge types before and after the second stage of fine-tuning.**
| Origin Type | New Type | Num |
| --- | --- | --- |
| HighlyKnown | HighlyKnown | 47335 |
| MaybeKnown | 2527 |  |
| WeaklyKnown &Unknown | 97 |  |
| MaybeKnown | HighlyKnown | 6131 |
| MaybeKnown | 13161 |  |
| WeaklyKnown & Unknown | 1373 |  |
| WeaklyKnown & Unknown | HighlyKnown | 225 |
| MaybeKnown | 2600 |  |
| WeaklyKnown & Unknown | 95715 |  |

## 5 Discussion

We proposed that fine-tuning can alter the classification of knowledge that was not directly involved in the fine-tuning process, and we conducted experiments to validate this hypothesis. Based on this insight, we refined the fine-tuning process, which led to improvements in both model accuracy on the test set and the model’s mastery of the knowledge in the fine-tuned dataset. However, our experiments remain largely qualitative, focusing primarily on demonstrating the feasibility of this approach.

Compared to the specific domain datasets that might be used for fine-tuning large language models in real-world applications, the WikiQA dataset we used is relatively loosely structured. This dataset consists of simple question-answer pairs with a limited set of question formats, and the questions lack the strong internal coherence typically found in domain-specific datasets. In such specialized datasets, data is often more focused within a particular domain, where the tighter connections between knowledge might lead to more significant changes in knowledge classification during fine-tuning.

Moreover, for knowledge that falls outside the scope of closed-book question-answering tasks, determining whether the knowledge is present in the model is more challenging. These differences indicate that our approach has broader potential applications but also faces new challenges.

Furthermore, when conducting continual learning, effectively addressing catastrophic forgetting requires a balanced approach between new and old data. Better application of continual fine-tuning methods is necessary to achieve this balance. In our experiments, we only utilized basic data replay and moderately reduced the learning rate, indicating that there is still significant room for improvement.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: - In writing, please add a space between all citations - it becomes hard to read the full sentences!
- What is the overall recommendation in terms of finetuning for multiple stages? It is not clear from the text.
soundness: 2
strengths: - Conducts several ablations on sequential multi-stage finetuning on easy to hard dataset subsets.
summary: The paper proposes a two stage finetuning strategy which supposedly improves the models knowledge retention capacity. Using the taxonomies introduced by [1], the paper conducts multi-stage finetuning ablations on the different sequence of subsets.

[1] https://arxiv.org/pdf/2405.05904
weaknesses: - The contributions of the paper is minimal, given they performed ablations on data taxonomies introduced by [1]. At best, the work is an extension of the ablations performed by [1] themselves (see Section 5 of the paper).
- Only one downstream dataset (WikiQA) is considered, which limits the broader applicability of the approach.
- The paper is very poorly written, with large swaths of the text borderline unintelligible. Specially sections 3.1, 3.2, 4.1.2 need a lot of work to make the flow understadable.
- Numerous evidence on poor quality of writing / organization: Table on page 5 is redundant, statistical analysis on changes from one subset to the next is not highlighted well in Tables 2,3,4,12. It is highly unclear what I'm looking at - the change should be presented _relatively_, not as absolute numbers which is meaningless.   



[1] https://arxiv.org/pdf/2405.05904

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Maybe the author can help answer the second weakness.
soundness: 3
strengths: 1. Efforts in designing strategies for how we fine-tuning with the replay data and the training data. We can see that, some unknown data change to MaybeKnown, justifying the  hypothesis.
2. Retain the model original performance, and also comparable performance with SFT over new data (1-stage fine-tuning).
summary: This paper attempts to address the knowledge forgetting problem in the scenario of continual fine-tuning, where we might lost some information/performance if we do continual fine-tuning. 

They perform 2-stage fine-tuning:
1. Categorize the data into certain categories to understand how the model knows about the previous knowledge.
2. The second stage with mix up with some replay data to continue fine-tuning.
weaknesses: 1. Paper is not well written, especially the citation position. 
2. It’s not practical to assume we have the original replay data. For example, if we use a pre-trained model (Qwen), they will not release the SFT data for the community. if we want to improve the model performance on QA using our new data, we probably don’t have original SFT data for replay. The assumption seems not practical.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - The compiled \cite looks broken throughout the paper.
soundness: 3
strengths: The motivation of this paper is well articulated, and it is important indeed such as how to avoid hallucination and catastrophic forgetting when LLMs are fine-tuned after pre-training.
summary: This paper tackles the challenges that LLMs primarily acquire knowledge during pretraining and often struggle to integrate new information effectively in later stages such as fine-tuning and inference times. This can lead to issues like hallucinations and decreased performance (catastrophic forgetting).
The core argument of this paper is that fine-tuning LLMs on partially mastered knowledge can help them leverage their existing knowledge base and reasoning capabilities to comprehend new concepts. The authors propose a two-stage fine-tuning to test this hypothesis.
Stage 1: The model is fine-tuned on data representing knowledge it partially understands, meaning it has some chance of answering related questions correctly without fine-tuning.
Stage 2: The training data is augmented with data that demonstrates improved mastery after the first stage, including knowledge initially classified as less well-understood. This augmented dataset is then used for a second round of fine-tuning.
The results on the WikiQA dataset show that the two-stage fine-tuning method leads to improved test accuracy, enhanced knowledge mastery and mitigation of forgetting.
weaknesses: 1. Limited generalizability due to focus on a single closed-book question answering: The experiments rely heavily on the WikiQA dataset, which is specifically designed for closed-book question answering. This focus raises concerns about the generalizability of the results to other knowledge domains and tasks. 

2. Tiny Incremental Improvements: While the two-stage fine-tuning method shows improvements in test accuracy and knowledge mastery compared to one-stage fine-tuning, these improvements are relatively small. For example, Table 7 in the source shows that for the Qwen2 model, the accuracy improvement from one-stage to two-stage is smaller than 1%. This raises the question of whether the added complexity and computational cost of the second stage are justified by such marginal gains.

3. Practicality of the method: Given the need for pre-classification, the practicality of the two-stage method for tasks that are not already determined is debatable.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Formatting:

Missing spaces and inconsistent citation format throughout the paper (e.g., Intro lines 40, 45, 51).

Terminology:

Define terms before using them:
“KG” is used without explanation (line 67).
Knowledge types (line 75).
Prompt template (line 247).
The second stage of finetuning in the Introduction is unclear and would benefit from clarification.

Methodology:

- Method Section 3.2: clarify data categories used in the finetuning steps (lines 203-206).
- Confirm if knowledge re-detection after each finetuning stage is performed on the training set—currently unclear.
- Definitions and Supplementary Information:
   Table 1 is identical to the one in Gekhman’s work (Figure 2(a)), and presented in the main paper which might be mistakenly interpreted as 
   some contribution.
- Define “Origin” in Table 7. Does it represent baseline performance without finetuning on ‘MaybeKnown’?
- Consider adding a supplementary section, e.g., for explaining prompt template creation (line 247).

Results Interpretation:

- Table 8: Strategy 1 and Strategy 4 have identical scores—does this imply an effect specific to ‘HighlyKnown’ knowledge?
Divide the main results from analyses.
soundness: 1
strengths: - Proposes Interesting intuition: integrating partially-known knowledge can boost performance on new known knowledge
        - Presents a clear, structured methodology in Fig. 1
summary: The paper investigates the challenges of finetuning in language models, noting that introducing new knowledge can lead to hallucinations and degraded performance. The authors hypothesize that reinforcing partially mastered knowledge could improve the model’s performance with new knowledge. They use a two-phase approach: (1) Knowledge detection (types) and (2) Two-stage finetuning with 'MaybeKnown' and 'HighlyKnown' Knowledge Replay. Primarily using Qwen-2, they analyze the contributions of each finetuning stage.

Writing and Clarity:

  - The paper suffers from unclear writing, especially in the abstract and motivation sections.
  - Poor organization and coherence in presenting experiment motivation, methods, and results, impacting the overall flow.
  - Findings are not clear and are not connected to the observations enough.


Reliance on Previous Contribution:

    - Misinterpret ‘MaybeKnown’ recommendation (line 374) for improving performance on less known knowledge -  Gekhman et al 2024 claimed that the ‘MaybeKnown’ helps to less hallucinate data the model already knows.

Explanation and Motivation:
 
 Lacks clear motivation for choosing specific strategies:

    - No justification for preferring a replay-based approach for catastrophic forgetting (line 154).
    - Insufficient explanation for focusing on the ‘MaybeKnown’ category over ‘HighlyKnown’ for finetuning.
    - Weak or missing rationale for strategies 1-5 (lines 403-416).
    - Justification is absent for selecting the first answer in multi-answer QA (lines 245-246).
   - Foundational intuition (Section 3.1) for acquiring new knowledge from partial knowledge is weak and lacks references.


Weak Results:
  - The proposed method shows modest improvements (~4-7%) in accuracy (Table 7) which is not strongly associated with a specific factor.
  - Results lack consistency: replay strategies sometimes decrease performance (e.g., Table 9, Strategies 4 and 5 for WeaklyKnown).
  - Small performance changes in Table 8 might result from various, unexamined factors.
  - No analysis of performance differences between models (Qwen and LLaMA).
- Not robust - used only one dataset (closed).


Results and Clarity:
  - Results are presented in raw counts (Tables 3, 4, 5, 10), without proportions, making the transition and interpretation difficult.
  - Graph construction (line 338) is unclear and does not clearly support the hypothesis; the contribution of node reclassification is ambiguous.
weaknesses: The paper's contribution appears limited, as the concept of using ‘MaybeKnown’ knowledge to enhance test-set performance has already been established by Gekhman (2024). The newly proposed two-stage finetuning with a replay method results in only minor improvements (~1-2%), which, coupled with weak, non-robust analysis and disorganized writing, supports a recommendation for rejection.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 1
questions: I suggest the authors submit this paper maybe tot a workshop.
soundness: 1
strengths: * knowledge probing/ augmentation is an important research problem for large language models.
summary: This paper first designed an experiment to validate their hypothesis that as the model’s capacity to utilize existing knowledge improves during fine-tuning, previously unmastered knowledge may become more understandable. Then they propose a two-stage fine-tuning method to address this.
weaknesses: * The presentation in this paper is very poor. Actually, it's very hard for me to follow them due to the poor logic in the abstract and introduction. Also, figures and tables are not well-designed.

* Many format problems. There is no space between text and citation; cited and citep are not properly used in this paper; Some citations are even raw text (line 242)

* The proposed method and findings are trivial. And the experiment settings that tune the model in the test set are also problematic to me.