HFT: Half Fine-Tuning for Large Language Models

## Abstract

Abstract Large language models (LLMs) with one or more fine-tuning phases have become a necessary step to unlock various capabilities, enabling LLMs to follow natural language instructions or align with human preferences.
However, it carries the risk of catastrophic forgetting during sequential training, the parametric knowledge or the ability learned in previous stages may be overwhelmed by incoming training data.
In this paper, we find that by regularly resetting partial parameters, LLMs can restore some of the original knowledge.
Inspired by this,
we introduce H alf F ine- T uning (HFT) for LLMs, as a substitute for full fine-tuning (FFT), to mitigate the forgetting issues, where half of the parameters are selected to learn new tasks while the other half are frozen to remain previous knowledge.
We provide a feasibility analysis from the perspective of optimization and interpret the parameter selection operation as a regularization term. Without changing the model architecture, HFT could be seamlessly integrated into existing fine-tuning frameworks.
Extensive experiments and analysis on supervised fine-tuning, direct preference optimization, and continual learning consistently demonstrate the effectiveness, robustness, and efficiency of HFT.
Compared with FFT, HFT not only significantly alleviates the forgetting problem, but also achieves the best performance in a series of downstream benchmarks, with an approximately 30% reduction in training time.

## 1 Introduction

Large language models (LLMs) bring immense revolutions to various natural language processing applications with powerful language understanding and generation capabilities.
Unsupervised large-scale pre-training for learning basic world knowledge (hereinafter referred to as basic knowledge), followed by one or more fine-tuning phases with supervised data or human feedback, is becoming a new training paradigm in the era of LLMs (Ouyang et al., ; Achiam et al., ; Touvron et al., ).
As the fine-tuning phase proceeds, the enormous potential of LLMs is gradually unleashed to handle various downstream tasks, while the parametric knowledge previously learned and stored in the pre-trained model might face a considerable risk of catastrophic forgetting (Lin et al., ; Neeman et al., ; Dong et al., ).
To maintain intrinsic basic knowledge, the most straightforward idea is to keep the pre-trained parameters unchanged and include extra modules to learn task-specific abilities (Dou et al., ; Wu et al., ). However, such modifications to the architecture pose significant obstacles to model deployment and continual fine-tuning.

Without changing model architecture, vanilla full fine-tuning (FFT) methods update all parameters to improve the performance of downstream tasks (Zhang et al., ), in which the element-wise parameter difference between fine-tuned and pre-trained models (i.e., task vector) represents the knowledge shift during fine-tuning (Ilharco et al., ).
Herein, a desirable task vector is expected to keep basic knowledge of the pre-trained model and learn new specialized knowledge.
Interestingly, recent work shows that partial dropping or trimming of the task vector has only milder impacts on target task (Yadav et al., ; Yu et al., ).
In other words, partial new parameters are sufficient for the learning of new abilities, so the upcoming question is, is it possible that a portion of old parameters could maintain the capabilities of the pre-trained model?

To answer this question, we start with Llama 2-7b and Llama 2-Chat-7b, and attempt to reset partial parameters of the chat-model to the pre-trained model, then prob the general abilities and basic knowledge of these models (see Figure 1).
As a representative general-purpose fine-tuning practice, there is some improvement in the general abilities of Llama 2-Chat-7b, while the basic knowledge falls off a cliff. It is consistent with previous observations, indicating the destruction of parametric knowledge stored in Llama 2-7b (Dou et al., ).
To balance the emerging general abilities and the inherent basic knowledge, we intuitively select and reset half of the parameters(^1^11Here, we keep the embedding and lm_head layers unchanged as Llama 2-Chat-7b, and select 50% of the parameters in transformer layers. The parameter ratios in this paper all follow this statistical calibre.) of Llama 2-Chat-7b and are pleasantly surprised to find that the Half-Reset model greatly resumes the basic knowledge in Llama 2-7b while remaining the excellent general abilities of Llama 2-Chat-7b (More details in Section 2).

Figure: Figure 1: Performance of Llama 2-7b, Llama 2-chat-7b, and the *Half-Reset* model on six general abilities and three basic knowledge benchmarks. It is interesting that simply resetting half of the parameters of the chat-model to the pre-trained model could roughly restore a significant amount of forgotten basic knowledge while maintaining high-level general abilities performance.

Inspired by these above observations, we propose Half Fine-Tuning (HFT), a simple yet effective approach for the training of LLMs and further extrapolate it to the continual fine-tuning scenarios.
Specifically, in each round of fine-tuning, we randomly select and freeze half of the parameters, and only update the other half, which allows the model to retain the ability of startup point while learning downstream tasks.
Note that HFT does not change the model architecture or traditional fine-tuning paradigm, thus theoretically it can be applied to any setting where the standard full fine-tuning is previously applicable, including but not limited to supervised fine-tuning (SFT), direct preference optimization (DPO), continual learning (CL), etc.

To evaluate the effectiveness of HFT in instruction fine-tuning settings, we conduct extensive experiments with Tülu V2 for SFT and UltraFeedback (Cui et al., ) for DPO.
Simultaneously, we also extend experiments on TRACE (Wang et al., ) for multi-round fine-tuning (i.e. continual learning), to validate the proposed method in a more extreme scenario.
Experimental results demonstrate that HFT not only exhibits excellent talent in alleviating catastrophic forgetting, but also achieves comparable or even better performance in learning new abilities compared to FFT.
Further analysis reveals that regardless of which half (or even only about half) of the parameters are selected, HFT is capable of attaining tolerable performance gains and impressive efficiency improvements, which brings considerable competition to the routine fine-tuning paradigm.

In summary, the main contributions of this paper are as follows:

- •
We reveal that by resetting half of the fine-tuned parameters to the startup state, it is possible to preliminary restore the primeval ability while maintaining new learning ability, which poses new opportunities to alleviate catastrophic forgetting and obtain an all-around LLM.
- •
We propose Half Fine-Tuning (HFT), which entails freezing half of the parameters while training the other half. It allows LLMs to acquire new abilities while retaining and utilizing previously learned knowledge in various training settings.
- •
Extensive experiments and analysis demonstrate the effectiveness and efficiency of HFT. Without any alterations to the model architecture, HFT, as a plug-and-play solution with only a few lines of code, exhibits the potential to supersede FFT in the era of LLMs.

## 2 Pilot Experiments

Considering that the partial task vector is capable of maintaining new abilities (Yadav et al., ; Yu et al., ), we attempt to roll back the primeval abilities of pre-trained models by resetting the remaining part of the task vector, thereby alleviating the catastrophic forgetting problem caused by fine-tuning.
In this section, We employ the representative well-aligned LLM, Llama 2-chat-7b, and the corresponding pre-trained backbone, Llama 2-7b, as models for analysis.

#### Setup.

To balance the original abilities and the enhanced capabilities gained through instruction tuning, we simply choose to reset 50% of the parameters in Llama 2-chat-7b to Llama 2-7b, so that half of the parameters are hoped to align with the new tasks, while the other half is intended to restore the old capabilities.
In implementation, we randomly select half of each transformer layer according to the category of parameter matrix. Specifically, we choose two from four self-attention matrices (i.e., $\mathbf{W}_{Q}$, $\mathbf{W}_{K}$, $\mathbf{W}_{V}$, $\mathbf{W}_{O·}$), and for the odd parameter number in Llama’s feed-forward layers (i.e., $\mathbf{W}_{up}$, $\mathbf{W}_{down}$, $\mathbf{W}_{gate}$), we randomly select half of the transformer layers to choose two matrices and the other half to choose one. Such a fine-grained selection strategy ensures that the *Half-Reset* operation rolls back exactly 50% of the parameters.

To assess the performance of the pre-trained, chat, and half-reset models on both new and old capabilities, we follow Ivison et al. () and Dou et al. () to introduce two categories of evaluation benchmarks: (1) General Abilities, including MMLU, GSM8K, BBH, TyDiQA, TruthfulQA, and HumanEval, which measure the LLMs’ newly enhanced abilities of performing specific downstream tasks like examination, reasoning, and coding. (2) Basic Knowledge, including NaturalQuestion, TriviaQA, and HotpotQA, which reflect the parametric world knowledge in the pre-trained model and could be used to evaluate retention of the primaeval capabilities.
For more details about the datasets and evaluation metrics, please refer to Appendix A.1 and A.2

#### Results.

From Figure 1, it is intuitive to observe significant improvement of Llama 2-chat-7b on several general ability benchmarks, as well as the comprehensive decline on the basic knowledge benchmarks.
When selectively restoring half parameters to the pre-trained Llama 2-7b model, although there is a slight performance loss in the average performance of general abilities, we witness the remarkable recovery of basic knowledge.
In Appendix B.1, we attempt other possible half-reset solutions and provide more numerical results, all of which exhibit similar phenomena.

As a conclusion, the pilot experiments demonstrate that (1) full parameter fine-tuning with large-scale instruction data disrupts the basic knowledge stored within pre-trained LLMs.
(2) Through a simple half-reset operation, it is possible to partially restore the forgotten knowledge.
*Take another step forward, these findings open a new door for model merging, inspiring us to preserve some mastered abilities of the startup point by freezing partial parameters during fine-tuning*.

## 3 Methodology

Without loss of generality, we consider a sequential (continual) learning setting with multiple tasks $\mathcal{T}$, in which each task corresponds to a set of input-output pairs $\mathcal{D}^{t}=\left\{x_{n}^{t},y_{n}^{t}\right\}_{n=1}^{\mathcal{N}^{t}}$.
In the training process, a single model aligns all the tasks sequentially, with only access to the specific dataset $\mathcal{D}^{t}$ at $t$-th round.
Formerly, given a LLM parameterized by $\theta$, the entire process aims to optimize the following objective, which encompasses all the tasks,

$$ $\small\mathcal{J}\left(\theta\right)=\max\limits_{\theta}\sum\limits_{t\in\{1, |\mathcal{T}|\}}{\sum\limits_{(x_{n}^{t},y_{n}^{t})\in\mathcal{D}^{t}}{\log{ \mathbf{P}_{\theta^{t}}\left(y_{n}^{t}|x_{n}^{t}\right)}}},$ (1) $$

where $\log\mathbf{P}(\cdot)$ represents the probability distribution of the model’s output. When there is only one task, the learning process degenerates into the standard supervised fine-tuning (SFT) form.

#### Half Fine-Tuning.

Next, we accordingly propose Half Fine-Tuning (HFT) to learn the upcoming new task while maintaining and utilizing old abilities.
Figure 2 illustrates the overall workflow of HFT, regarding the intermediate repetitive transformer layers, we divide each layer into three blocks: self-attention, feed-forward, and layernorm, so as half of each block is selected for updating in this round, while the remaining half is frozen.
Note that the frozen and updated parameters vary among each training round. In this way, HFT is more conducive to maintaining relative knowledge parity across different rounds during the sequential alignment process, thus exhibiting significant scalability in successive training.
From the formula perspective, we define the parameters that remain unchanged during the $t$-th round as $\psi^{t}$, and correspondingly, the parameters that align to the upcoming tasks as $\vartheta^{t}$ (i.e., $\theta^{t}=\{\vartheta^{t},\psi^{t}\}$). The training objective in Equation 1 thus changes to

$$ $\displaystyle\mathcal{J}\left(\theta\right)$ $\displaystyle=\max\limits_{\theta}\sum\limits_{t\in\{1,|\mathcal{T}|\}}{\sum \limits_{(x_{n}^{t},y_{n}^{t})\in\mathcal{D}^{t}}{\log{\mathbf{P}_{\{\vartheta ^{t},\psi^{t}\}}\left(y_{n}^{t}|x_{n}^{t}\right)}}},$ (2) $\displaystyle s.t.$ $\displaystyle\quad\vartheta^{t}\leftarrow\vartheta^{t-1}-\eta\nabla_{\vartheta }\mathcal{L}\left(\theta^{t-1}\right)~{},\quad\psi^{t}\leftarrow\psi^{t-1}~{},$ $$

where $\eta$ and $\mathcal{L}(\cdot)$ represent the learning rate and loss function, $\nabla_{\vartheta}$ indicates that we only consider the gradients of selected parameters in fine-tuning.

Figure: Figure 2: The schematic procedure of HFT with Llama 2’s architecture. In each stage, we selectively freeze half of the parameters at the category-level and update the other half. Best viewed in color.

#### Why Half Fine-Tuning Works.

Excluding heuristic motivations, we are also interested in the theoretical principles behind HFT. Theoretically, HFT could be regarded as exerting a parameter-level mask to vanilla FFT. In this part, we borrow the thread in Fu et al. () to interpret why HFT works from the perspective of optimization.
Given a pre-trained model $\mathcal{M}^{0}$ with parameters $\theta^{0}$, the fine-tuned model $\mathcal{M}$ with parameters $\theta$ has the same structure as $\mathcal{M}^{0}$ such that $\|\theta-\theta^{0}\|_{0}\leq p\dim(\theta)$, where $p=0.5$ in HFT.
Next, we denote $M\in\{0,1\}^{m\times m}$ as a mask diagonal matrix on the parameter, in which the diagonal is equal to $1$ if the parameter is selected, thus the fine-tuning procedure can be formulated as $\theta=\theta^{0}+M\Delta\theta$, where $\Delta\theta$ is the task vector.
In that case, HFT solves an optimization problem with constraints
$\min_{\Delta\theta,M}\mathcal{L}(\theta^{0}+M\Delta\theta)$ such that $\|M\|_{0}=\lfloor mp\rfloor$; $M_{ij}=0,\;\forall i\neq j$; $M_{ii}\in\{0,1\}.$
where $\mathcal{L}$ is the loss function, $\lfloor\cdot\rfloor$ is the floor function, $m$ is the parameter numbers.
By integrating previous conditions, the optimization procedure of HFT can be reformulated as

$$ $\small\mathcal{O}=\min_{\theta}\mathcal{L}(\theta)\quad s.t.\ \ \|(I-M)(\theta -\theta^{0})\|^{2}=0,$ (3) $$

With Lagrangian duality, solving the constrained optimization problem is equivalent to solving the following unconstrained optimization problem

$$ $\small\mathcal{O}_{L}=\min_{\theta}\max_{\lambda}\mathcal{L}(\theta)+\lambda\| (I-M)(\theta-\theta^{0})\|^{2},$ (4) $$

where $\lambda$ is the Lagrange multiplier.
Based on the Minimax inequality, it is intuitive to derive that
$\min_{\theta}\max_{\lambda}\mathcal{L}(\theta)+\lambda\|(I-M)(\theta-\theta^{0
})\|^{2}\geq\max_{\lambda}\min_{\theta}\mathcal{L}(\theta)+\lambda\|(I-M)(
\theta-\theta^{0})\|^{2}\geq\min_{\theta}\mathcal{L}(\theta)+\|(I-M)(\theta-
\theta^{0})\|^{2}$.
In conclusion, the optimization process of HFT is equivalent to optimizing the upper bound of the FFT loss function $\mathcal{L}(\theta)$ with a regularization term $\|(I-M)(\theta-\theta^{0})\|^{2}$.
From the optimization perspective, such regularization (with an appropriate sparsity $M$) contributes to the stability of the sparse fine-tuned model (Radiya-Dixit and Wang, ; Fu et al., ), meaning that HFT has the opportunity to achieve results comparable to or even better than FFT, theoretically.

## 4 Experiments

In this section, we primarily report the experimental results of full fine-tuning (FFT) and the proposed half fine-tuning (HFT) on supervised fine-tuning (with Tülu V2 (Ivison et al., ) as training set), human preference alignment (with UltraFeedback (Cui et al., )), and continual learning (with TRACE (Wang et al., )) scenarios, in which direct preference optimization (DPO) (Rafailov et al., ) is used to learn human preferences.
Following Ivison et al. () and Wang et al. (), we employ Llama 2 and Llama 2-chat as the backbone model, respectively.
For more comprehensive understanding of implementation details, please refer to Apendix A.

**Table 1: Results on general ability benchmarks of various models with instruction tuning (SFT, DPO), in which the default setting is FFT, R and H refer to the proposed Half-Reset and Half Fine-Tuning methods, respectively. Bold text denotes the best result in each group.**
|  | MMLU | GSM8K | BBH | TyDiQA | TruthfulQA | HumanEval | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| (factuality) | (reasoning) | (reasoning) | (multilingual) | (truthful) | (coding) |  |  |
|  | EM | EM | EM | F1 | MC2 | Pass@10 |  |
| (0-shot) | (8-shot, CoT) | (3-shot, CoT) | (1-shot, GP) | (0-shot) | (0-shot) |  |  |
| Pre-trained models |  |  |  |  |  |  |  |
| Llama 2-7b | 41.6 | 12.0 | 39.9 | 48.4 | 38.5 | 26.2 | 34.4 |
| Llama 2-13b | 52.2 | 34.5 | 50.7 | 50.3 | 49.8 | 32.7 | 45.0 |
| Supervised Fine-Tuning (SFT) on Tülu V2 |  |  |  |  |  |  |  |
| Llama 2-7b-SFT | 48.5 | 25.0 | 42.2 | 51.2 | 41.7 | 36.9 | 41.0 |
| Llama 2-7b-SFT (R) | 48.4 | 23.0 | 43.4 | 52.4 | 42.5 | 32.5 | 40.4 |
| Llama 2-7b-SFT (H) | 50.8 | 30.5 | 43.6 | 52.3 | 45.4 | 34.6 | 42.9 (+1.9) |
| Llama 2-13b-SFT | 50.6 | 45.0 | 47.8 | 55.0 | 42.6 | 42.4 | 47.2 |
| Llama 2-13b-SFT (R) | 52.7 | 46.0 | 52.8 | 55.5 | 46.8 | 41.4 | 49.2 |
| Llama 2-13b-SFT (H) | 54.5 | 46.5 | 53.7 | 56.7 | 45.7 | 43.5 | 50.1 (+2.9) |
| Direct Preference Optimization (DPO) on UltraFeedback |  |  |  |  |  |  |  |
| Llama 2-7b-DPO | 48.9 | 28.0 | 42.9 | 50.2 | 45.7 | 35.6 | 41.9 |
| Llama 2-7b-DPO (R) | 49.0 | 28.5 | 43.1 | 50.3 | 43.3 | 34.8 | 41.5 |
| Llama 2-7b-DPO (H) | 48.8 | 25.5 | 42.8 | 51.1 | 45.5 | 36.7 | 41.7 (-0.2) |
| Llama 2-13b-DPO | 52.0 | 44.0 | 47.1 | 51.5 | 45.5 | 44.3 | 47.4 |
| Llama 2-13b-DPO (R) | 51.5 | 46.5 | 48.2 | 53.7 | 43.7 | 42.7 | 47.7 |
| Llama 2-13b-DPO (H) | 51.8 | 48.5 | 49.9 | 52.9 | 45.3 | 41.0 | 48.2 (+0.8) |

### 4.1 Experiments on Instruction Tuning

#### Setup.

We employ the general abilities and basic knowledge benchmarks mentioned in Section 2 to evaluate various models under the instruction tuning settings. To assess the conversation ability, we also compare these models on AlpacaEval 2.0 (see also Appendix B.2).

#### Results on Improving General Abilities.

Results in Table 1 demonstrate the effectiveness of our proposed HFT method, which simultaneously improves different specialized abilities by selectively fine-tuning half of the parameters.
Specifically, compared to FFT under the SFT setting, HFT leads to an average performance improvement of 1.9% on Llama 2-7b and 2.9% when scaling to Llama 2-13b.
Furthermore, as we continue to perform DPO on SFT models, we observe that updating the policy model with HFT does not hinder the model from learning human preferences.
In sum, the HFT method has strong robustness to adapt to different fine-tuning algorithms.
Besides, we also attempt to review the Half-Reset method in Section 2, but the benefits of this approach are not robust, and we attribute it to the randomness of parameter operations.
In comparison, HFT achieves a more stable performance improvement through the learning process, while avoiding the complexity of the two-stage process of fully updating followed by partially resetting.

#### Results on Preserving Basic Knowledge.

For the basic knowledge benchmarks, as depicted in Table 2, both SFT and DPO exhibit a significant decline across all three benchmarks.
In contrast, our proposed HFT method consistently outperforms both fine-tuning and alignment processes in the evaluation of basic knowledge.
Notably, HFT demonstrates excellent talent in preserving basic knowledge, consistently outperforming fully updating parameters during SFT and DPO.
For example, in the SFT stage, HFT achieves improvements of 3.4% and 2.9% with Llama 2-7b and Llama 2-13b compared to FFT, respectively.
Besides, it is also worth mentioning that the Half-Reset method shows a relatively stable performance in alleviating knowledge forgetting.

Remark. HFT not only effectively preserves a certain degree of basic knowledge of the pre-trained model, but also utilizes this knowledge to achieve better learning of new abilities.

**Table 2: Results on basic knowledge benchmarks of various models with instruction tuning.**
|  | NaturalQuestion | TriviaQA | HotpotQA | Average |
| --- | --- | --- | --- | --- |
| (EM, 0-shot) | (EM, 0-shot) | (EM, 0-shot) |  |  |
| Pre-trained models |  |  |  |  |
| Llama 2-7b | 12.9 | 40.2 | 15.6 | 22.9 |
| Llama 2-13b | 9.6 | 24.0 | 13.4 | 15.7 |
| Supervised Fine-Tuning (SFT) on Tülu V2 |  |  |  |  |
| Llama 2-7b-SFT | 3.2 | 26.4 | 14.5 | 14.7 |
| Llama 2-7b-SFT (R) | 7.3 | 26.4 | 14.4 | 16.0 |
| Llama 2-7b-SFT (H) | 6.2 | 32.8 | 15.4 | 18.1 (+3.4) |
| Llama 2-13b-SFT | 0.7 | 9.2 | 4.9 | 4.9 |
| Llama 2-13b-SFT (R) | 1.8 | 13.5 | 5.3 | 6.9 |
| Llama 2-13b-SFT (H) | 2.7 | 12.4 | 8.2 | 7.8 (+2.9) |
| Direct Preference Optimization (DPO) on UltraFeedback |  |  |  |  |
| Llama 2-7b-DPO | 1.4 | 20.8 | 10.0 | 10.7 |
| Llama 2-7b-DPO (R) | 2.0 | 23.6 | 12.1 | 12.6 |
| Llama 2-7b-DPO (H) | 1.9 | 22.9 | 12.8 | 12.5 (+1.8) |
| Llama 2-13b-DPO | 0.1 | 4.4 | 2.4 | 2.3 |
| Llama 2-13b-DPO (R) | 0.3 | 6.5 | 3.8 | 3.5 |
| Llama 2-13b-DPO (H) | 0.2 | 5.5 | 3.0 | 2.9 (+0.6) |

### 4.2 Experiments on Continual Learning

#### Setup.

We assess the performance in the continual learning scenario (with TRACE (Wang et al., )), using four representative approaches and attempt to replace FFT with HFT.
(1) SeqFT: It is a standard for sequentially learning all parameters of downstream tasks.
(2) GEM (Lopez-Paz and Ranzato, ): It leverages episode memories to avoid forgetting, but it consumes extra computation time like other regularization-based methods.
(3) Replay: It is a common continual learning strategy, here we integrate alignment data from LIMA (Zhou et al., ) into the replay memory and replaying 10% of historical data.
(4) LoraSeqFT (Hu et al., ): It sequentially updates the low-rank matrices while keeping the backbone fixed. Note that the LoRA-based method modifies the model architecture and is not suitable for combination with HFT.
Following (Wang et al., ), we start with Llama 2-chat-7b/13b, adopt Overall Performance (OP) and Backward Transfer (BWT) as the evaluation metrics (Appendix A.2 details the calculation process).
Besides, we also report the general abilities and basic knowledge of various models after the final round of learning (see Appendix B.4).

**Table 3: OP and BWT on TRACE with different strategies, OP measures the learning of new tasks and BWT measures the forgetting of old tasks.**
|  | FFT | HFT |  |  |
| --- | --- | --- | --- | --- |
| OP | BWT | OP | BWT |  |
| Llama 2-chat-7b |  |  |  |  |
| LoraSeqFT | 6.4 | -45.2% | - | - |
| SeqFT | 45.7 | -10.2% | 51.3 (+5.6) | -5.6% (+4.6) |
| GEM | 48.2 | -7.9% | 50.2 (+2.0) | -5.9% (+2.0) |
| Replay | 54.3 | 1.4% | 54.1 (-0.2) | +2.1% (+0.7) |
| Llama 2-chat-13b |  |  |  |  |
| LoraSeqFT | 26.5 | -30.0% | - | - |
| SeqFT | 49.0 | -9.4% | 52.0 (+3.0) | -8.5% (+0.9) |
| GEM | 50.4 | -8.9% | 53.6 (+3.2) | -6.1% (+2.8) |
| Replay | 54.7 | -0.6% | 57.4 (+2.7) | +1.6% (+2.2) |

#### Results.

Table 3 shows that the three FFT approaches could all benefit from equipping HFT.
Specifically, HFT brings performance improvements of 5.7% and 2.0% on the OP metric in the SeqFT and GEM settings, respectively. It also boosts the performance with 4.6%, 0.7%, and 2.0% on the BWT metric based on the Llama 2-chat-7b.
When scaling the model to 13b, HFT could also achieve superior performances.
Further, fine-tuning with full parameters often suffers from severe catastrophic forgetting in the $5$-th round (see Appendix B.8), while HFT does not experience such a problem in any of the rounds, making the learning process more stable.
Besides, LoraSeqFT exhibits notably suboptimal performance in this setting. We assume that the knowledge capacity of LoRA parameter is quite limited, thus resulting in considerable forgetting during the process of sequential training.
On the contrary, HFT is based on a full set of parameters and selects half of the parameters to be fine-tuned in each round, which has stronger knowledge tolerance.

Remark. HFT is naturally suitable for scenarios with continual fine-tuning, and (almost all) methods with FFT can be further improved by assembling HFT, highlighting the plug-and-play feature.

### 4.3 Impact of Parameter Selection

HFT heuristically selects parameters to be tuned or frozen. We hope to reveal the impact of parameter selection from parameter radio and selection strategy, to discuss the universality of the methodology.

#### Impact of Trainable Parameter Ratio.

Firstly, we traverse the radio of parameters to be fine-tuned at a granularity of $\sim$10% and evaluate the impact in both single-round and multi-round fine-tuning scenarios.
From Figure 3, we observe that most of the results with only updating partial parameters are superior to FFT, and the performance is quite satisfactory when the trainable parameter radio is around 50%.
In SFT, the performance of basic knowledge shows a clear downward trend with the increase of parameter ratio, while the general abilities slowly rise, which allows updating half or less of the parameters to have good performance.
Meanwhile, when selecting half of the parameters during continual learning, the model reaches a balance of abilities between each round of tasks, resulting in a more robust training procedure and optimal performance.
This observation again confirms the early conjecture about catastrophic forgetting, especially in continual learning, it is necessary to freeze a portion of parameters in each round to preserve the capabilities of the previous models.
Not only that, we also find that fixing partial parameters gradually improves training efficiency (see Table 11), and HFT could shorten the training time by 30% in standard SFT.

Figure: (a) Performance of models trained on Tülu V2.

**Table 4: Different strategies for selecting half of the parameters on TRACE.**
|  | OP | BWT |
| --- | --- | --- |
| SeqFT (FFT) | 45.7 | -10.2% |
| SeqFT (Model-level HFT) | 46.9 (+1.2) | -9.2% (+1.0%) |
| SeqFT (Layer-level HFT) | 47.9 (+2.2) | -8.3% (+1.9%) |
| SeqFT (Category-level HFT) | 51.3 (+5.6) | -5.6% (+4.6%) |

#### Impact of Selection Strategy.

Next, we consider other possible strategies for selecting half of the parameters:
(1) Model-level. It arbitrarily chooses half the number of parameter matrices, which may prevent the parameter ratio from accurately reaching 50%.
(2) Layer-level. It selects all parameters of a layer every other layer.
(3) Category-level. It selects based on parameter categories, which is the default strategy used in this paper, and ensures the accurate selection of 50% of the parameters.
Table 4 reports the results of performing HFT on TRACE with sequential fine-tuning (SeqFT).
The first noteworthy phenomenon is that all three selection strategies outperform the standard FFT, which once again confirms the motivation that freezing some parameters helps balance the old and new abilities in continual fine-tuning.
Moreover, the category-level selection wins the best performance, we attribute it to the fine-grained strategy that maximizes the interaction between updated and non-updated parameters.
From the perspective of model merging, it minimizes the damage to ready-made capabilities when performing a 50% dropout on the task vector, thereby providing greater possibilities for learning new tasks based on existing knowledge.

Remark. HFT is robust and insensitive to parameter selection, and selecting approximately 50% of the parameters with a reasonable selection strategy could achieve acceptable improvements.

**Table 5: General abilities and basic knowledge performance of HFT models fine-tuned on Tülu V2 without embedding (E) and lm_head (H) layers. Specially, the subscript indicates the proportion of selected parameters to the total model parameters.**
|  | MMLU | GSM | BBH | TyDi | Truthful | Human | Natural | Trivia | Hotpot | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 8K | QA | QA | Eval | Questions | QA | QA |  |  |  |
| ${\rm\text{HFT}_{41.2\%}}$ (freeze E,H) | 49.9 | 26.0 | 44.6 | 52.3 | 45.0 | 33.2 | 6.3 | 24.0 | 14.1 | 32.8 |
| ${\rm\text{HFT}_{51.9\%}}$ (freeze E,H) | 50.8 | 30.5 | 43.6 | 52.3 | 45.4 | 34.6 | 6.2 | 32.8 | 15.4 | 34.6 |
| ${\rm\text{HFT}_{48.0\%}}$ (update E,H) | 51.4 | 29.0 | 45.0 | 50.5 | 45.2 | 35.0 | 3.2 | 24.1 | 13.7 | 33.0 |

Figure: (a) Variations on SAN in various transformer blocks.

## 5 Discussion

In this section, we further discuss the parameter changes in the fine-tuning process to deepen the understanding of HFT.
We review the influence of embedding and lm_head layers, and visualize the parameter variations during successive training.

#### Revisit the Embedding and LM_head Layers.

**Table 6: OP and BWT scores of HFT models fine-tuned on TRACE without embedding and lm_head layers.**
|  | OP | BWT |
| --- | --- | --- |
| ${\rm\text{HFT}_{41.2\%}}$ (freeze E,H) | 49.6 | -5.6% |
| ${\rm\text{HFT}_{51.9\%}}$ (freeze E,H) | 51.3 | -5.6% |
| ${\rm\text{HFT}_{48.0\%}}$ (update E,H) | 46.1 | -2.2% |

HFT defaults to updating the embedding and lm_head layers. Here, we aim to explore the roles of these two input and output layers.
Specifically, we freeze them while maintaining the same selection strategy and report results in supervised fine-tuning and continual learning.
Since freezing the embedding and lm_head layers slightly reduces trainable parameters, we also include another model with similar parameter ratios that only freeze the parameters in transformer layers, to mitigate the impact of parameter ratio.
As shown in Table 5, freezing these two layers leads to a substantial decline in knowledge-intensive benchmarks, especially for QA-related tasks.
Experimental results in Table 6 witness another phenomenon, where forgetting metric BWT significantly increases while the learning metric OP faces a cliff-like decrease.
Detailed results in Appendix B.6 reveal that there is a substantial decline in the performance of ScienceQA.
To this extent, a preliminary conjecture emerges that the embedding and lm_head store information are highly relevant to world knowledge, so it is crucial to update them during the fine-tuning process.

#### Parameters Variation Analysis.

To intuitively perceive the difference in model parameters between HFT and FFT, we visualize parameter variations of fine-tuned models relative to the initial model (Llama 2-Chat-7b) during continual learning on TRACE.
On the one hand, we group two adjacent layers and calculate the average variation of self-attention and feed-forward blocks, where average variation refers to the average of all matrix differences in the block of two models.
On the other hand, based on the selected number of times in these eight rounds of fine-tuning, we compare the average variation of each block with FFT.
Figure 4 shows variations from the perspective of transformer block and selected time, respectively.
Interestingly, we find that:
(1) The parameter variation of each layer using HFT is fainter than those using FFT.
(2) There is no significant difference in parameter variation between shallow and deep transformer layers, which is consistent in both fine-tuning settings.
(3) The deviation from pre-trained parameters increases linearly with the time of selection, and the variations of parameters selected eight times are very similar to FFT.
Therefore, excessive offset of task vectors may not necessarily lead to an improvement in downstream performance, but result in forgetting existing capabilities. HFT seeks subtle balance by pulling back the task vector, alleviating catastrophic forgetting when learning subsequent tasks.

## 6 Related Work

#### Sparse Fine-tuning.

With the continuous increase in the number of language model parameters, sparse fine-tuning (a.k.a. parameter-efficient fine-tuning (PEFT)) offers an effective solution by reducing trainable parameters while achieving comparable performance to FFT (Fu et al., ; Ding et al., ; Han et al., ).
Adapter (Houlsby et al., ; Mahabadi et al., ; Zhang et al., ) and LoRA (Hu et al., ; Dou et al., ; Dettmers et al., ), the two most famous kinds of work, freeze the initial model weight and inject an adapter or a trainable rank decomposition matrices into each layer.
However, these approaches change the model architecture and therefore require customized deployment.
Keeping the architecture unchanged, DiffPruning (Guo et al., ) learns a sparse diff vector for each task and enables PEFT scales well with new tasks.
BitFit (Zaken et al., ) only fine-tunes the bias terms of BERT and achieves considerably good performance.
Unfortunately, these methods designed for specific tasks or networks (e.g., bias) are not suitable for modern general-purpose large-scale models.
Recently, some work proposes to fuse various abilities through the arithmetic of task vectors (Ilharco et al., ; Xiao et al., ; Yu et al., ).
Inspired by these findings, HFT further considers how to retain the pre-trained knowledge without adding additional modules, and incorporates the idea of model merging into the training process.

#### Continual Learning.

Continual learning aims to develop learning algorithms that can accumulate knowledge on non-stationary data, and vanilla FFT has been proven to lead to severe catastrophic forgetting issues when adapting to incoming streaming tasks (Luo et al., ; Wang et al., ).
To address this issue,
experience replay (Rolnick et al., ; Peng et al., ) is a widely used technique that incorporates a portion of data from previous rounds into the current training process.
Regularization-based models (Kirkpatrick et al., ; Lopez-Paz and Ranzato, ) introduce additional terms in the loss function to penalize changes in crucial weights.
Parameter-allocation approaches (Li et al., ; Gurbuz and Dovrolis, ) feature an isolated parameter subspace dedicated to each task throughout the network.
When LLMs enter the era of billions of parameters,
researchers prefer to use progressive prompts (Razdaibiedina et al., ) or PEFT (Dou et al., ; Wu et al., ) to tune a powerful general backbone for specific tasks or domains (Wu et al., ).
Instead of introducing auxiliary modules or losses, HFT explores a new direction based on the characteristics of LLMs, proving that random parameter selection is sufficient to achieve passable performance and has the potential to become a successor to FFT.

## 7 Conclusion

In this paper, we observe that rolling back half of the fine-tuned parameters to the pre-trained state may recover partial knowledge of the startup model while holding the performance of downstream tasks.
Taking inspiration from this observation, we propose Half Fine-tuning (HFT), which adopts a category-level strategy to select half of the parameters for updating in each training round, and the remaining parameters are expected to maintain the learned knowledge.
Extensive experiments on the supervised fine-tuning, direct preference optimization, and continual learning scenarios demonstrate the effectiveness of HFT. It not only alleviates the catastrophic forgetting in preceding capabilities but also achieves comparable or even more superior performance than FFT in downstream tasks.
Further analysis shows that HFT is robust to selection strategies and selected parameter numbers.
Last but not least, HFT does not change the model architecture, making it easy to implement and scale, especially under successive fine-tuning scenarios.

## Appendix A Experimental Setup

### A.1 Datasets

To validate the performance of supervised fine-tuning, we choose Tülu V2 [Ivison et al., ] which is a combination of high-quality open resources, including datasets (1) created by researchers from existing NLP datasets (e.g. SuperNI [Wang et al., ]), (2) written by humans (e.g. Dolly [Conover et al., ] and Open Assistant [Köpf et al., ]), (3) generated by LLMs (e.g. Self-Instruct [Wang et al., ], Alpaca [Taori et al., ] and Baize [Xu et al., ]), (4) comprised of user-shared prompts accompanied by model-generated completions (e.g. ShareGPT [Chiang et al., ]), and (5) developed for specific abilities (e.g. CoT [Wei et al., ] for chain-of-thought and Code-Alpaca [Chaudhary, ] for code generation).
To examine the capacity for reinstating a fraction of impaired capabilities while adhering to human preferences, we utilize UltraFeedback [Cui et al., ] which is a large-scale, high-quality, and diversified preference dataset.
For continual learning, we select TRACE [Wang et al., ], a novel benchmark designed for continual learning (CL) in LLMs, to evaluate catastrophic forgetting in standard CL settings. TRACE consists of 8 distinct datasets spanning challenging tasks, domain-specific tasks, multilingual capabilities, code generation, and mathematical reasoning.

### A.2 Evaluation Metrics

#### Supervised Fine-Tuning and Direct Preference Optimization.

To validate the effectiveness of our method, we employ general abilities and basic knowledge benchmarks to assess the performance in learning new tasks and preserving the original capabilities, respectively.
Specifically, for the general abilities benchmarks, we include the following evaluation sets to test various abilities. (1) Factual knowledge: To assess the LLMs’ factual knowledge, we employ the Massive Multitask Language Understanding dataset (MMLU) [Hendrycks et al., ]. MMLU comprises a collection of questions across 57 subjects from elementary to professional difficulty levels. We report the 5-shot accuracy based on answer perplexity. (2) Reasoning: We utilize the test split of the Grade School Math (GSM8K) dataset [Cobbe et al., ] and Big-Bench-Hard (BBH) [Suzgun et al., ] to evaluate the reasoning abilities. We report the 8-shot accuracy and the exact match (EM) rates for GSM8K and BBH, respectively. (3) Multilingualism: To evaluate multilingual capabilities, we employ TyDiQA [Clark et al., ], a multilingual question-answering benchmark that covers 11 typologically diverse languages. We adopt the gold-passage setup, where a passage containing the reference answer is provided, and report the F1 score. (4) Coding: To evaluate the LLMs’ ability to generate functionally correct programs from docstrings, we utilize the HumanEval dataset [Chen et al., ] and report the pass@10 performance. (5) Truthful: We incorporate TruthfulQA [Lin et al., ] to assess the ability to avoid generating known falsehoods resulting from misconceptions or false beliefs while providing informative responses.
(6) LLM-based evaluation: We use AlpacaEval 2.0 [Li et al., ] to evaluate the instruction-following abilities. AlpacaEval is an LLM-based automatic evaluation metric. In this paper, we calculate the win rates against the GPT-4-preview-1106.
We include the following three datasets for basic knowledge benchmarks to validate the basic knowledge preserved in LLMs: (1) NaturalQuestion [Kwiatkowski et al., ], (2) TriviaQA [Han et al., ], and (3) HotpotQA [Yang et al., ]

#### Continual Learning.

For continual learning evaluations, following [Wang et al., ], we use Overall Performance (OP) and Backward Transfer (BWT) scores as the main metrics in CL settings. In terms of the formula, after incrementally learning the $t$-th task, the performance on the $i$-th task (where $i\leq t$) is denoted as $S_{t,i}$. The OP and BWT scores can be calculated as

$$ $\small{\rm OP}_{t}=\frac{1}{t}\sum_{i=1}^{t}S_{t,i},\quad{\rm BWT}_{t}=\frac{1 }{t}\sum_{i=1}^{t-1}\left(S_{t,i}-S_{i,i}\right).$ (5) $$

We utilize accuracy as the primary evaluation metric for C-STANCE, FOMC, ScienceQA, NumGLUE-cm, and NumGLUE-ds. In the case of Py150, we employ similarity as the evaluation metric. Moreover, for the evaluation of MeetingBank and 20Minuten, we employ the ROUGE-L metric.

Figure: Algorithm 1 Algorithm of HFT with Category-Leval Parameter Selection

**Table 7: General abilities and basic knowledge results of Llama 2-7b, the well-aligned model Llama 2-chat-7b, and our proposed three half-reset approaches.**
|  | Llama 2- | Llama 2- | Model-level | Layer-level | Category-level |
| --- | --- | --- | --- | --- | --- |
|  | 7b | chat-7b | Half-Reset | Half-Reset | Half-Reset |
| MMLU (EM, 0-shot) | 41.6 | 47.0 | 46.2 | 45.8 | 46.7 |
| GSM (ACC, 8-shot) | 12.0 | 26.0 | 8.0 | 22.0 | 24.0 |
| BBH (EM, 0-shot) | 39.9 | 39.2 | 41.0 | 39.5 | 37.7 |
| TyDiQA (F1, 1-shot) | 48.4 | 43.6 | 46.3 | 44.2 | 44.9 |
| TruthfulQA (MC2, 0-shot) | 38.5 | 46.0 | 41.7 | 43.1 | 41.7 |
| HumanEval (Pass@10) | 26.2 | 23.9 | 26.8 | 25.0 | 22.0 |
| General abilities average | 34.4 | 37.6 | 35.0 | 36.6 | 36.2 |
| NaturalQuestion (EM, 0-shot) | 12.9 | 7.2 | 8.2 | 11.2 | 10.9 |
| TriviaQA (EM, 0-shot) | 40.2 | 3.3 | 18.3 | 21.3 | 21.3 |
| HotpotQA (EM, 0-shot) | 15.6 | 6.6 | 7.4 | 9.9 | 9.0 |
| World knowledge average | 22.9 | 5.7 | 11.3 | 12.4 | 13.7 |
| Overall average | 30.6 | 27.0 | 27.1 | 28.5 | 28.7 |

### A.3 Implementation Details

Following [Ivison et al., ], in the SFT phase on Tülu V2, we adopt a linear-decreasing learning rate of 2e-5 with a 0.3 warmup ratio and train for 2 epochs.
For the human preference alignment phase on UltraFeedback, we use direct preference optimization [Rafailov et al., ] to align the fine-tuned LLMs on Tülu V2. We use a learning rate of 5e-7 and a global batch size of 32.
Due to the context length of 4096 used during Llama 2 pre-training, as referenced in the [Ivison et al., ] code repository issues, we set a maximum sequence length of 4096 during the SFT stage. However, due to hardware resource limitations, the maximum sequence length is reduced to 1024 during the DPO stage under Llama 2-13b.
During the continual learning phase, following [Wang et al., ], we employ a fixed learning rate of 1e-5 and fine-tune the eight sub-datasets for different numbers of epochs: 5, 3, 7, 5, 3, 5, 5, and 7 epochs, respectively. The global batch size for both stages is set to 128.
All our experiments are conducted on one machine equipped with 8x80G Nvidia A100.
Algorithm 1 introduce the detailed implementations of our proposed fine-grained selecting approach of HFT.
Additionally, to evaluate the SFT and DPO models, we employ a chat format, using specialized tokens 
 and 
 to mark user utterances and target assistant responses, respectively. However, for HumanEval and the basic knowledge benchmarks, we use a standard language format when evaluating pre-trained models.

## Appendix B Additional Experiments

### B.1 Detailed Results of Pilot Experiments

Table 7 presents the detailed results of the experiments conducted in Section 2. We compare two additional parameter selection methods, Random and Alternate. We apply these methods to the reset process. The results indicate that the category-level selection approach achieves the highest average performance, which is consistent with the conclusion in follow-up training settings.

### B.2 Evaluation on AlpacaEval

**Table 8: Evaluation results on AlpacaEval 2.0.**
| Models | AlpacaEval 2.0 |
| --- | --- |
| Llama 2-7b-SFT | 6.96 |
| Llama 2-7b-SFT (R) | 2.98 |
| Llama 2-7b-SFT (H) | 5.59 |
| Llama 2-7b-DPO | 10.68 |
| Llama 2-7b-DPO (R) | 8.44 |
| Llama 2-7b-DPO (H) | 9.07 |
| Llama 2-13b-SFT | 8.32 |
| Llama 2-13b-SFT (R) | 11.93 |
| Llama 2-13b-SFT (H) | 10.43 |
| Llama 2-13b-DPO | 11.55 |
| Llama 2-13b-DPO (R) | 12.55 |
| Llama 2-13b-DPO (H) | 11.68 |

As shown in Table 8, we evaluate different models on AlpacaEval 2.0. The results indicate that our method is less effective than FFT on Llama 2-7b. However, a reversal occurs when the model size scales up to 13b, where our approach outperforms the FFT models comprehensively. This suggests that our method has greater potential on much larger-scale LLMs, as supported by the experimental results in Table 1, which show a larger improvement of HFT compared to FFT on Llama 2-13b compared to Llama 2-7b.
Interestingly, the Half-Reset method performs well on Llama 2-13b but shows completely different results on Llama 2-7b. This suggests that simply resetting half of the parameters may not provide consistent performance since the model is trained on the full set of parameters.

**Table 9: General abilities and basic knowledge performance of DPO stage (with HFT), which is initialized with HFT-based SFT models fine-tuned on Tülu V2.**
|  | MMLU | GSM | BBH | TyDi | Truthful | Human | Natural | Trivia | Hotpot | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 8K | QA | QA | Eval | Question | QA | QA |  |  |  |  |
| DPO (FFT-based, 7b) | 48.8 | 25.5 | 42.8 | 51.1 | 45.5 | 36.7 | 1.9 | 22.9 | 12.8 | 32.0 |
| DPO (HFT-based, 7b) | 50.7 | 30.5 | 42.8 | 43.9 | 49.8 | 35.1 | 1.0 | 20.4 | 5.9 | 31.1 |
| DPO (FFT-based, 13b) | 51.8 | 48.5 | 49.9 | 52.9 | 45.3 | 41.0 | 0.2 | 5.5 | 3.0 | 33.1 |
| DPO (HFT-based, 13b) | 55.0 | 45.5 | 51.4 | 53.2 | 49.5 | 42.9 | 0.3 | 4.9 | 4.7 | 34.2 |

**Table 10: General abilities and basic knowledge performance of the final round models fine-tuned on TRACE. We compare four different fine-tuning methods and our HFT approach on Llama 2-chat-7b and Llama 2-chat-13b.**
|  | MMLU | GSM | BBH | TyDi | Truthful | Human | Natural | Trivia | Hotpot | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 8K | QA | QA | Eval | Question | QA | QA |  |  |  |  |
| SeqFT-7b | 35.5 | 3.0 | 24.3 | 39.1 | 42.7 | 0.3 | 10.0 | 23.9 | 14.0 | 21.4 |
| GEM-7b | 40.1 | 3.5 | 17.0 | 33.4 | 41.4 | 2.2 | 10.0 | 19.6 | 14.0 | 20.1 |
| Replay-7b | 45.9 | 4.5 | 35.2 | 41.6 | 39.6 | 8.5 | 11.6 | 36.1 | 14.2 | 26.4 |
| LoraSeqFT-7b | 43.3 | 11.0 | 30.7 | 35.5 | 41.7 | 8.8 | 8.7 | 24.7 | 13.4 | 24.2 |
| SeqFT-7b (H) | 44.1 | 3.5 | 30.8 | 41.1 | 41.8 | 1.6 | 11.3 | 38.9 | 14.4 | 25.3 (+3.9) |
| GEM-7b (H) | 45.1 | 5.0 | 32.3 | 34.9 | 43.0 | 2.7 | 10.4 | 35.9 | 13.7 | 24.8 (+4.7) |
| Replay-7b (H) | 47.9 | 11.0 | 38.8 | 42.6 | 42.5 | 12.7 | 10.7 | 38.4 | 12.9 | 28.6 (+2.2) |
| SeqFT-13b | 39.7 | 5.0 | 27.9 | 41.0 | 41.4 | 0.0 | 12.7 | 44.3 | 16.3 | 25.4 |
| Replay-13b | 49.0 | 3.5 | 40.1 | 37.7 | 43.1 | 12.0 | 12.5 | 6.7 | 13.3 | 24.2 |
| GEM-13b | 47.2 | 4.0 | 37.6 | 36.3 | 43.0 | 10.0 | 10.8 | 10.2 | 12.1 | 23.5 |
| LoraSeqFT-13b | 43.3 | 15.0 | 42.4 | 43.1 | 40.5 | 18.2 | 10.6 | 37.6 | 16.2 | 29.7 |
| SeqFT-13b (H) | 50.0 | 7.0 | 46.3 | 47.2 | 41.4 | 11.2 | 14.7 | 50.6 | 18.7 | 31.9 (+6.5) |
| GEM-13b (H) | 49.9 | 9.5 | 46.5 | 38.2 | 45.1 | 18.9 | 9.8 | 39.7 | 14.2 | 30.2 (+6.7) |
| Replay-13b (H) | 50.0 | 10.5 | 47.1 | 39.6 | 45.8 | 20.1 | 10.1 | 41.1 | 14.0 | 30.9 (+2.3) |

**Table 11: Efficiency analysis among different ratios of trainable parameters.**
| Trainable Parameters (%) | 8.3% | 22.3% | 30.6% | 38.9% | 50.0% | 61.1% | 69.4% | 77.7% | 91.7% | 100% |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Runtime (%) | 48.0% | 52.2% | 56.4% | 64.0% | 68.5% | 72.5% | 85.1% | 85.2% | 89.0% | 100% |
| $\Delta$ | -52.0 | -47.8 | -43.6 | -36.0 | -31.5 | -27.5 | -14.9 | -14.8 | -11.0 | 0.0 |

### B.3 Direct Preference Optimization with HFT-based Models

In Section 4.1, we initialize our DPO process with the FFT model. In this section, we investigate the performance of the DPO process when initialized with the HFT model. The experimental results are presented in Table 9. We observe that while the DPO process on the HFT model performs better in certain general abilities, such as TruthfulQA, it experiences minor losses in overall performance under Llama 2-7b. However, the situation is reversed in Llama 2-13b, where the DPO deployed on the HFT model outperforms the FFT-initialized DPO.
Nonetheless, DPO equipped with HFT tends to consistently improve performance compared to DPO with FFT.

### B.4 General Abilities and Basic Knowledge of Continual Fine-tuned Model

We also evaluate the models mentioned in Section 4.2 on general abilities and basic knowledge benchmarks. The experimental results are presented in Table 10. We observe that after 8 rounds of fine-tuning on consecutive tasks, the models fine-tuned with the HFT method consistently outperform the FFT models in terms of average performance. This further confirms the effectiveness of HFT in preserving the original capabilities of the model and mitigating catastrophic forgetting.
Furthermore, although LoRA preserves more layer parameters unchanged, it still performs worse compared to HFT. We believe this may be attributed to the low-rank decomposition resulting in a limited number of trainable parameters. Merging the LoRA weights back into the original model could potentially disrupt the original parameter space to a greater extent.

### B.5 Efficiency Analysis

We conduct a comparison of the runtime costs for different ratios of trainable parameters. Specifically, we fine-tuned Llama 2-7b on Tülu V2 and record the total duration from the start to the end of the training program. The results in Table 11 demonstrate that, without specific optimization, all models with varying ratios of trainable parameters can reduce the training time. As expected, as the proportion of trainable parameters increases, the training duration also increases. Notably, our HFT method achieves a 31.5% reduction in training time, significantly decreasing the training cost for extremely large-scale instruction datasets.

### B.6 Detailed Results of Revisiting Embedding and LM_Head Layers

Table 12 details the results of freezing the input and output layers.
Meanwhile, Table 13 and 14 show the detailed results of the two adjacent numbers of parameter settings on TRACE.

**Table 12: Detailed results on TRACE without updating embedding and lm_head layers.**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 50.1 | 48.0 | 47.2 | 45.8 | 46.4 | 46.2 | 46.3 | 48.0 |
| FOMC | - | 69.0 | 66.1 | 65.7 | 65.7 | 64.7 | 63.9 | 66.9 |
| MeetingBank | - | - | 37.5 | 34.5 | 34.2 | 32.7 | 31.9 | 33.2 |
| Py150 | - | - | - | 51.2 | 50.3 | 49.8 | 49.2 | 50.8 |
| ScienceQA | - | - | - | - | 58.1 | 58.0 | 56.8 | 56.2 |
| NumGLUE-cm | - | - | - | - | - | 33.3 | 25.9 | 29.6 |
| NumGLUE-ds | - | - | - | - | - | - | 45.8 | 43.1 |
| 20Minuten | - | - | - | - | - | - | - | 40.6 |
| Average | 50.1 | 58.5 | 50.3 | 49.3 | 50.9 | 47.5 | 45.7 | 46.1 |
| BWT | - | - | - | - | - | - | - | -2.2% |

### B.7 Detailed Results of Different Parameter Selection Strategies

Table 15 and 16 provide the detailed results of model-level and layer-level parameter selection strategies mentioned in Section 4.3.

**Table 13: Detailed results on TRACE with 41.24% trainable parameters.**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 49.2 | 43.7 | 43.2 | 44.2 | 44.2 | 44.4 | 43.7 | 45.1 |
| FOMC | - | 71.0 | 64.3 | 65.3 | 60.7 | 65.9 | 65.1 | 63.3 |
| MeetingBank | - | - | 46.9 | 37.7 | 35.4 | 39.0 | 38.5 | 36.9 |
| Py150 | - | - | - | 57.9 | 52.6 | 53.6 | 53.6 | 53.4 |
| ScienceQA | - | - | - | - | 85.7 | 77.5 | 71.8 | 74.8 |
| NumGLUE-cm | - | - | - | - | - | 33.3 | 29.6 | 33.3 |
| NumGLUE-ds | - | - | - | - | - | - | 56.6 | 48.9 |
| 20Minuten | - | - | - | - | - | - | - | 41.1 |
| Average | 49.2 | 57.4 | 51.5 | 51.3 | 55.7 | 52.3 | 51.3 | 49.6 |
| BWT | - | - | - | - | - | - | - | -5.6% |

**Table 14: Detailed results on TRACE with 62.65% trainable parameters.**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 45.3 | 50.8 | 50.9 | 51.4 | 51.3 | 51.4 | 51.1 | 53.3 |
| FOMC | - | 72.8 | 63.7 | 65.7 | 6.3 | 68.3 | 69.0 | 67.9 |
| MeetingBank | - | - | 48.9 | 41.1 | 38.3 | 41.3 | 41.1 | 40.0 |
| Py150 | - | - | - | 57.3 | 50.3 | 52.8 | 52.9 | 52.9 |
| ScienceQA | - | - | - | - | 88.2 | 70.6 | 67.3 | 69.4 |
| NumGLUE-cm | - | - | - | - | - | 30.9 | 28.4 | 21.0 |
| NumGLUE-ds | - | - | - | - | - | - | 59.4 | 53.5 |
| 20Minuten | - | - | - | - | - | - | - | 40.8 |
| Average | 45.3 | 61.8 | 54.5 | 53.9 | 46.9 | 52.6 | 52.7 | 49.9 |
| BWT | - | - | - | - | - | - | - | -5.6% |

**Table 15: Detailed results on TRACE with model-level parameter selection.**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 49.3 | 49.1 | 48.8 | 50.2 | 50.0 | 48.9 | 48.1 | 49.2 |
| FOMC | - | 70.6 | 57.5 | 53.8 | 42.7 | 54.4 | 58.1 | 55.2 |
| MeetingBank | - | - | 48.9 | 37.8 | 36.5 | 38.2 | 37.3 | 38.9 |
| Py150 | - | - | - | 57.7 | 55.4 | 55.9 | 54.8 | 55.7 |
| ScienceQA | - | - | - | - | 87.7 | 59.8 | 54.2 | 56.4 |
| NumGLUE-cm | - | - | - | - | - | 38.3 | 22.2 | 25.9 |
| NumGLUE-ds | - | - | - | - | - | - | 55.7 | 53.5 |
| 20Minuten | - | - | - | - | - | - | - | 40.7 |
| Average | 49.3 | 59.9 | 51.7 | 49.9 | 54.5 | 49.3 | 47.2 | 46.9 |
| BWT | - | - | - | - | - | - | - | -9.2% |

**Table 16: Detailed results on TRACE with layer-level parameter selection.**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 50.8 | 41.4 | 44.6 | 46.5 | 47.5 | 48.6 | 48.2 | 49.0 |
| FOMC | - | 72.2 | 58.5 | 54.6 | 1.8 | 46.8 | 50.2 | 50.0 |
| MeetingBank | - | - | 47.1 | 34.7 | 34.5 | 37.2 | 38.6 | 37.1 |
| Py150 | - | - | - | 56.5 | 53.3 | 53.8 | 54.2 | 54.1 |
| ScienceQA | - | - | - | - | 88.5 | 84.4 | 76.2 | 77.5 |
| NumGLUE-cm | - | - | - | - | - | 35.8 | 28.4 | 21.0 |
| NumGLUE-ds | - | - | - | - | - | - | 57.2 | 52.9 |
| 20Minuten | - | - | - | - | - | - | - | 41.5 |
| Average | 50.8 | 56.8 | 50.1 | 48.1 | 45.1 | 51.1 | 50.4 | 47.9 |
| BWT | - | - | - | - | - | - | - | -8.3% |

### B.8 Detailed Results of TRACE

Table 17 to 30 show the detailed results of different models and approaches of each round during the continual learning on TRACE.

**Table 17: Detailed results on TRACE with SeqFT (start from Llama 2-chat-7b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 48.5 | 49.7 | 48.5 | 48.3 | 6.7 | 47.4 | 47.2 | 48.7 |
| FOMC | - | 71.6 | 46.6 | 46.4 | 0.4 | 43.1 | 42.9 | 44.0 |
| MeetingBank | - | - | 49.0 | 39.9 | 40.8 | 37.6 | 34.5 | 37.9 |
| Py150 | - | - | - | 57.0 | 49.2 | 54.5 | 54.2 | 54.0 |
| ScienceQA | - | - | - | - | 89.1 | 71.5 | 44.6 | 60.6 |
| NumGLUE-cm | - | - | - | - | - | 30.9 | 24.7 | 25.9 |
| NumGLUE-ds | - | - | - | - | - | - | 59.4 | 52.6 |
| 20Minuten | - | - | - | - | - | - | - | 41.5 |
| Average | 48.5 | 60.7 | 48.0 | 47.9 | 37.2 | 47.5 | 43.9 | 45.7 |
| BWT | - | - | - | - | - | - | - | -10.2% |

**Table 18: Detailed results on TRACE with SeqFT and HFT (start from Llama 2-chat-7b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 49.4 | 47.6 | 45.6 | 46.4 | 47.8 | 49.5 | 49.1 | 49.3 |
| FOMC | - | 71.8 | 57.7 | 59.1 | 46.0 | 66.5 | 67.3 | 66.3 |
| MeetingBank | - | - | 47.4 | 39.1 | 31.2 | 38.6 | 38.4 | 35.7 |
| Py150 | - | - | - | 57.4 | 52.1 | 54.8 | 55.0 | 55.0 |
| ScienceQA | - | - | - | - | 87.4 | 82.1 | 77.6 | 75.3 |
| NumGLUE-cm | - | - | - | - | - | 42.0 | 30.9 | 32.1 |
| NumGLUE-ds | - | - | - | - | - | - | 58.5 | 55.1 |
| 20Minuten | - | - | - | - | - | - | - | 41.3 |
| Average | 49.4 | 59.7 | 50.2 | 50.5 | 52.9 | 55.6 | 53.8 | 51.3 |
| BWT | - | - | - | - | - | - | - | -5.6% |

**Table 19: Detailed results on TRACE with GEM (start from Llama 2-chat-7b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 50.0 | 48.9 | 48.4 | 47.7 | 13.0 | 46.5 | 45.7 | 48.1 |
| FOMC | - | 69.4 | 60.3 | 59.7 | 0.4 | 56.5 | 57.1 | 58.5 |
| MeetingBank | - | - | 49.0 | 40.4 | 38.4 | 38.8 | 34.8 | 39.0 |
| Py150 | - | - | - | 56.7 | 51.2 | 54.0 | 53.6 | 53.8 |
| ScienceQA | - | - | - | - | 89.5 | 64.2 | 29.5 | 54.5 |
| NumGLUE-cm | - | - | - | - | - | 33.3 | 32.1 | 33.3 |
| NumGLUE-ds | - | - | - | - | - | - | 59.7 | 57.2 |
| 20Minuten | - | - | - | - | - | - | - | 40.8 |
| Average | 50.0 | 59.2 | 52.6 | 51.1 | 38.5 | 48.9 | 44.6 | 48.2 |
| BWT | - | - | - | - | - | - | - | -7.9% |

**Table 20: Detailed results on TRACE with GEM and HFT (start from Llama 2-chat-7b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 50.3 | 49.0 | 47.0 | 48.3 | 50.0 | 50.7 | 50.1 | 51.3 |
| FOMC | - | 70.0 | 58.9 | 60.1 | 36.1 | 63.9 | 65.9 | 65.5 |
| MeetingBank | - | - | 47.5 | 40.2 | 38.2 | 39.2 | 39.0 | 37.9 |
| Py150 | - | - | - | 57.0 | 53.0 | 55.3 | 55.1 | 54.6 |
| ScienceQA | - | - | - | - | 88.4 | 76.8 | 70.1 | 68.4 |
| NumGLUE-cm | - | - | - | - | - | 34.6 | 24.7 | 29.6 |
| NumGLUE-ds | - | - | - | - | - | - | 60.0 | 53.6 |
| 20Minuten | - | - | - | - | - | - | - | 41.0 |
| Average | 50.3 | 59.5 | 51.1 | 51.4 | 53.1 | 53.4 | 52.1 | 50.2 |
| BWT | - | - | - | - | - | - | - | -5.9% |

**Table 21: Detailed results on TRACE with Replay (start from Llama 2-chat-7b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 51.7 | 50.1 | 49.4 | 48.2 | 50.6 | 49.7 | 49.9 | 52.0 |
| FOMC | - | 64.9 | 68.1 | 70.2 | 70.0 | 70.0 | 70.6 | 70.0 |
| MeetingBank | - | - | 43.4 | 48.0 | 46.1 | 46.5 | 46.4 | 44.8 |
| Py150 | - | - | - | 53.9 | 55.0 | 54.1 | 54.0 | 53.5 |
| ScienceQA | - | - | - | - | 81.9 | 86.0 | 86.3 | 87.5 |
| NumGLUE-cm | - | - | - | - | - | 30.9 | 32.1 | 32.1 |
| NumGLUE-ds | - | - | - | - | - | - | 55.7 | 53.5 |
| 20Minuten | - | - | - | - | - | - | - | 40.6 |
| Average | 51.7 | 57.5 | 53.6 | 55.1 | 60.7 | 56.2 | 56.4 | 54.3 |
| BWT | - | - | - | - | - | - | - | 1.4% |

**Table 22: Detailed results on TRACE with Replay and HFT (start from Llama 2-chat-7b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 47.7 | 53.5 | 50.6 | 51.0 | 50.8 | 50.2 | 51.1 | 52.1 |
| FOMC | - | 61.1 | 69.4 | 70.8 | 69.8 | 70.2 | 69.4 | 69.8 |
| MeetingBank | - | - | 39.3 | 47.1 | 47.0 | 46.0 | 46.7 | 47.3 |
| Py150 | - | - | - | 55.3 | 56.3 | 56.3 | 56.5 | 55.6 |
| ScienceQA | - | - | - | - | 87.3 | 52.2 | 85.0 | 84.8 |
| NumGLUE-cm | - | - | - | - | - | 37.0 | 29.6 | 32.1 |
| NumGLUE-ds | - | - | - | - | - | - | 48.0 | 50.5 |
| 20Minuten | - | - | - | - | - | - | - | 40.5 |
| Average | 47.7 | 57.3 | 53.1 | 56.1 | 62.2 | 52.0 | 55.2 | 54.1 |
| BWT | - | - | - | - | - | - | - | +2.1% |

**Table 23: Detailed results on TRACE with LoRASeqFT (start from Llama 2-chat-7b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 51.6 | 48.1 | 47.4 | 46.9 | 24.1 | 12.0 | 4.1 | 7.9 |
| FOMC | - | 68.8 | 58.3 | 52.6 | 0.0 | 48.4 | 44.2 | 1.4 |
| MeetingBank | - | - | 45.7 | 10.6 | 5.9 | 1.1 | 2.7 | 3.0 |
| Py150 | - | - | - | 58.6 | 20.8 | 46.8 | 45.2 | 0.4 |
| ScienceQA | - | - | - | - | 66.1 | 50.7 | 41.3 | 0.0 |
| NumGLUE-cm | - | - | - | - | - | 33.3 | 27.2 | 0.0 |
| NumGLUE-ds | - | - | - | - | - | - | 50.5 | 0.0 |
| 20Minuten | - | - | - | - | - | - | - | 38.1 |
| Average | 51.6 | 58.5 | 50.5 | 42.2 | 23.4 | 32.1 | 30.7 | 6.4 |
| BWT | - | - | - | - | - | - | - | -45.2% |

**Table 24: Detailed results of on TRACE with SeqFT (start from Llama 2-chat-13b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 51.3 | 34.9 | 37.6 | 40.0 | 41.0 | 44.2 | 43.8 | 44.9 |
| FOMC | - | 70.0 | 57.5 | 52.6 | 4.2 | 49.0 | 47.2 | 49.8 |
| MeetingBank | - | - | 50.5 | 44.9 | 44.4 | 45.7 | 44.7 | 41.9 |
| Py150 | - | - | - | 56.8 | 54.9 | 54.4 | 53.1 | 54.6 |
| ScienceQA | - | - | - | - | 91.3 | 73.5 | 66.1 | 73.9 |
| NumGLUE-cm | - | - | - | - | - | 43.2 | 28.4 | 25.9 |
| NumGLUE-ds | - | - | - | - | - | - | 62.5 | 59.4 |
| 20Minuten | - | - | - | - | - | - | - | 41.4 |
| Average | 51.3 | 52.5 | 48.5 | 48.6 | 47.2 | 51.7 | 49.4 | 49.0 |
| BWT | - | - | - | - | - | - | - | -9.4% |

**Table 25: Detailed results on TRACE with SeqFT and HFT (start from Llama 2-chat-13b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 54.2 | 52.2 | 54.7 | 55.2 | 55.3 | 54.3 | 54.6 | 55.5 |
| FOMC | - | 73.4 | 56.7 | 54.6 | 38.3 | 43.1 | 41.9 | 50.2 |
| MeetingBank | - | - | 48.9 | 44.4 | 44.1 | 45.5 | 45.9 | 43.6 |
| Py150 | - | - | - | 58.9 | 56.3 | 56.4 | 56.7 | 56.3 |
| ScienceQA | - | - | - | - | 89.7 | 84.3 | 74.5 | 74.6 |
| NumGLUE-cm | - | - | - | - | - | 54.3 | 33.3 | 35.8 |
| NumGLUE-ds | - | - | - | - | - | - | 64.0 | 59.4 |
| 20Minuten | - | - | - | - | - | - | - | 40.9 |
| Average | 54.2 | 62.8 | 53.4 | 53.3 | 56.7 | 56.3 | 53.0 | 52.0 |
| BWT | - | - | - | - | - | - | - | -8.5% |

**Table 26: Detailed results on TRACE with GEM (start from Llama 2-chat-13b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 51.5 | 47.2 | 46.7 | 48.1 | 19.0 | 47.4 | 48.3 | 49.2 |
| FOMC | - | 70.5 | 59.4 | 60.2 | 0.0 | 60.7 | 58.2 | 61.2 |
| MeetingBank | - | - | 52.3 | 47.6 | 40.5 | 40.6 | 43.2 | 41.5 |
| Py150 | - | - | - | 60.7 | 60.2 | 53.6 | 54.6 | 55.7 |
| ScienceQA | - | - | - | - | 92.7 | 78.5 | 30.6 | 60.5 |
| NumGLUE-cm | - | - | - | - | - | 43.7 | 33.3 | 33.3 |
| NumGLUE-ds | - | - | - | - | - | - | 61.7 | 60.2 |
| 20Minuten | - | - | - | - | - | - | - | 41.8 |
| Average | 51.5 | 58.9 | 52.8 | 54.2 | 42.5 | 54.1 | 47.1 | 50.4 |
| BWT | - | - | - | - | - | - | - | -8.9% |

**Table 27: Detailed results on TRACE with GEM and HFT (start from Llama 2-chat-13b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 52.4 | 51.5 | 48.9 | 49.6 | 51.5 | 51.0 | 50.2 | 51.5 |
| FOMC | - | 73.4 | 60.8 | 61.9 | 44.4 | 65.3 | 68.9 | 67.2 |
| MeetingBank | - | - | 50.2 | 47.6 | 41.2 | 43.3 | 40.9 | 41.8 |
| Py150 | - | - | - | 61.7 | 60.1 | 60.3 | 58.7 | 57.5 |
| ScienceQA | - | - | - | - | 93.0 | 88.7 | 78.9 | 77.7 |
| NumGLUE-cm | - | - | - | - | - | 44.4 | 33.3 | 36.7 |
| NumGLUE-ds | - | - | - | - | - | - | 61.9 | 55.7 |
| 20Minuten | - | - | - | - | - | - | - | 40.6 |
| Average | 52.4 | 62.5 | 53.3 | 55.2 | 58.0 | 58.8 | 56.1 | 53.6 |
| BWT | - | - | - | - | - | - | - | -6.1% |

**Table 28: Detailed results on TRACE with Replay (start from Llama 2-chat-13b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 48.8 | 51.3 | 48.5 | 49.3 | 49.2 | 47.5 | 46.7 | 51.4 |
| FOMC | - | 62.3 | 70.6 | 72.4 | 71.2 | 71.2 | 70.8 | 73.0 |
| MeetingBank | - | - | 44.9 | 48.2 | 47.4 | 48.5 | 47.1 | 47.5 |
| Py150 | - | - | - | 53.9 | 55.1 | 54.2 | 47.5 | 53.3 |
| ScienceQA | - | - | - | - | 89.5 | 91.6 | 90.7 | 89.6 |
| NumGLUE-cm | - | - | - | - | - | 45.7 | 29.6 | 30.9 |
| NumGLUE-ds | - | - | - | - | - | - | 57.5 | 52.3 |
| 20Minuten | - | - | - | - | - | - | - | 39.7 |
| Average | 48.8 | 56.8 | 54.7 | 56.0 | 62.5 | 59.8 | 55.7 | 54.7 |
| BWT | - | - | - | - | - | - | - | -0.6% |

**Table 29: Detailed results on TRACE with Replay and HFT (start from Llama 2-chat-13b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 50.2 | 52.5 | 53.8 | 53.0 | 53.4 | 52.7 | 52.4 | 52.1 |
| FOMC | - | 61.3 | 74.2 | 71.2 | 71.8 | 73.2 | 72.4 | 73.6 |
| MeetingBank | - | - | 48.5 | 48.7 | 47.0 | 46.9 | 48.6 | 47.6 |
| Py150 | - | - | - | 55.7 | 58.2 | 55.4 | 54.0 | 54.5 |
| ScienceQA | - | - | - | - | 83.3 | 90.0 | 90.1 | 89.7 |
| NumGLUE-cm | - | - | - | - | - | 45.7 | 48.1 | 43.2 |
| NumGLUE-ds | - | - | - | - | - | - | 60.9 | 57.5 |
| 20Minuten | - | - | - | - | - | - | - | 41.0 |
| Average | 50.2 | 56.9 | 58.8 | 57.2 | 62.7 | 60.7 | 60.9 | 57.4 |
| BWT | - | - | - | - | - | - | - | +1.6% |

**Table 30: Detailed results on TRACE with LoRASeqFT (start from Llama 2-chat-13b).**
| Task\Round | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C-STANCE | 52.4 | 44.4 | 45.1 | 39.0 | 0.0 | 41.8 | 41.1 | 12.4 |
| FOMC | - | 67.1 | 58.3 | 43.8 | 2.2 | 60.3 | 57.8 | 0.0 |
| MeetingBank | - | - | 47.3 | 11.3 | 18.2 | 14.6 | 3.2 | 12.2 |
| Py150 | - | - | - | 59.2 | 40.0 | 47.7 | 50.0 | 23.6 |
| ScienceQA | - | - | - | - | 75.4 | 70.3 | 71.0 | 67.7 |
| NumGLUE-cm | - | - | - | - | - | 47.5 | 28.5 | 25.7 |
| NumGLUE-ds | - | - | - | - | - | - | 61.3 | 28.6 |
| 20Minuten | - | - | - | - | - | - | - | 41.6 |
| Average | 52.4 | 55.8 | 50.2 | 38.3 | 27.2 | 47.0 | 44.7 | 26.5 |
| BWT | - | - | - | - | - | - | - | -30.0% |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why did the authors choose to update 50% of the parameters? Given the numerous possible combinations of 50% parameter selections, how do they determine which parameters to select?
2. What are the differences in the importance of each parameter when training with Direct Preference Optimization (DPO) compared to Supervised Fine-Tuning (SFT)? It would be insightful to explore methods for evaluating parameter importance rather than relying on a heuristic 50% selection.
3. Considering that DPO is typically used for alignment following SFT, what are the implications of applying parameter selection first on SFT and then on DPO?
4. How does the parameter selection strategy perform across different depths of network layers?
soundness: 2
strengths: 1. This work is well-organized and easy to follow.
2. The experimental results are comprehensive and demonstrate the effectiveness of updating a subset of parameters.
summary: This study introduces a parameter-efficient training method for fine-tuning large language models (LLMs) by limiting the number of model parameters that are updated (parameter isolation). The authors empirically choose to fine-tune 50% of the parameters and demonstrate that this approach can enhance the performance of Supervised Fine-Tuning, Direct Preference Optimization, and Continued Learning. They show that the model achieves better performance with 50% of the parameters updated compared to using all parameters.
weaknesses: Disadvantages:
From the view of a new fine-tuning methods, the design of the method is trivial. Numerous existing studies [1-4] have explored partial optimization (parameter isolation) to improve model performance. The heuristic selection of 50% of freezing parameters does not provide a robust framework for further advancements in the parameter selection domain. The so-called "half fine-tuning" method appears to be a specific instance of freezing selection or parameter isolation. Selection criteria could be the closeness to convergence (measure the gradient norm change between two timestamp)[1], the long-term weight history[2], block-wise periodical update history[3] and importance based soft layer sampling strategies[4].

The authors provide only limited analysis of partial parameter fine-tuning. The paper lacks a detailed examination of the importance of each layer and the choice of optimizer. Additionally, there is an absence of optimization design for parameter selection and strategies based on the selected parameters. Authors could try weight norm[4], the gradient norm change between two training steps[1] and the layer-wise parameter difference to make measurements.

In continual learning, parameter isolation [5-8] have been long-standing techniques. It would be beneficial to understand the specific differences when these are applied to LLMs. Authors could tell their methods uniqueness from previous ones by clarifying its special design compared with them and list the difference exactly for each previous methods like DERPP[5], progressive networks[6]. According to the current version, I would regard HFT as a special case of the mentioned ones.

[1] AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning, arXiv 2021

[2] SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezing, ICLR 2023

[3] BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models, arXiv 2024

[4] LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning, NeurIPS

[5] Dark experience for general continual learning: a strong, simple baseline, NeurIPS 2020

[6] Progressive neural networks, arXiv 2016

[7] Pathnet: Evolution channels gradient descent in super neural networks, arXiv 2017

[8] Packnet: Adding multiple tasks to a single network by iterative pruning, CVPR 2018

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: Please answer the questions mentioned in the weaknesses part.
soundness: 2
strengths: The method is simple, easy to implement, and can be generalized to any LLM. The experiments are sufficient and basically cover SFT and RLHF in addition to pre-training. Compared with full-parameter fine-tuning, the method proposed in this paper achieves better or comparable results.
summary: The authors consider a special form of fine-tuning LLMs, multi-step fine-tuning, where one task is fine-tuned at a time, and after multiple rounds of fine-tuning, the entire dataset is learned. This fine-tuning mode naturally suffers from the catastrophic forgetting problem. To address the catastrophic forgetting issue, the authors propose a method where half of the parameters are frozen during one fine-tuning session while the other half are tuned. The experimental results show that the method yields slight improvements in several scenarios, including supervised fine-tuning, human preference alignment, and normal continual learning.
weaknesses: The method of freezing a portion of parameters during fine-tuning is relatively common, and numerous studies have already validated the effectiveness of this freezing method. As a result, the proposed method appears a little incremental and lacks sufficient innovation. Furthermore, the effectiveness of the proposed method in the paper largely stems from the fact that the dataset size of each sub-task is relatively small. Fine-tuning a portion of parameters makes the model less prone to overfitting, rather than truly addressing the issue of catastrophic forgetting. Additionally, for LLMs, the continual learning setting, which trains only one sub-task at a time, seems rather toy-like. Such a specialized scenario is difficult to apply in practical real-world situations.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: See weakness
soundness: 3
strengths: 1. The paper is well-written and easy to follow

2. The proposed method is simple and effective

3. The experiments cover different settings
summary: This paper finds that LLMs can restore some of the original knowledge by regularly resetting partial parameters. Inspired by this, the authors introduce Half Fine-Tuning (HFT) for LLMs, as a substitute for full fine-tuning (FFT), to mitigate the forgetting issues, where half of the parameters are selected to learn new tasks and the other half are frozen to retain previous knowledge. Extensive experiments and analysis on supervised fine-tuning, direct preference optimization, and continual learning consistently demonstrate the effectiveness, robustness, and efficiency of HFT.
weaknesses: 1. Additional LLMs should be investigated to demonstrate the generalization ability of the proposed method.

2. In the continual learning setting, how does the proposed method compare to the baseline O-LoRA (Orthogonal Subspace Learning for Language Model Continual Learning)?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to my weakness part.
soundness: 2
strengths: The method is simple. The paper is easy to follow.
summary: This paper explores the problem of catastrophic forgetting in large language models (LLMs) that undergo sequential fine-tuning to enhance their ability to follow instructions and align with human preferences. Traditional full fine-tuning (FFT) can lead to the erosion of previously learned knowledge when new information is introduced. To address this, the authors propose Half Fine-Tuning (HFT), where only half of the model’s parameters are updated to learn new tasks, while the other half remain frozen, preserving earlier knowledge.
weaknesses: 1. **Lack of Distinct Advantage in PEFT Methodology**  
   - The paper proposes HFT as a new PEFT method but does not convincingly demonstrate its superiority over established alternatives like LoRA, Adapters, or BitFit. While the authors claim that "HFT allows LLMs to acquire new abilities while retaining and utilizing previously learned knowledge," this characteristic is inherent to all PEFT methods, not unique to HFT.
2. **Positioning of the "Half-Reset" Technique**  
   - The paper introduces the "Half-Reset" approach, yet its relevance and novelty are unclear. The method is shown to be less effective than HFT and is not new; parameter merging is already well-studied in the literature, with "Half-Reset" representing a specific case of selective parameter merging (merging at a 0.5 ratio) [1]. This technique does not constitute a meaningful contribution of the paper.
3. **Experimental Setup and Results**  
   - **Unclear Benchmark Setting**: The details of the TRACE benchmark setup lack clarity. It is unclear which tasks were used to evaluate continual learning baselines, and whether task IDs are accessible for training, given that methods like LoraSeqFT exhibit significant forgetting.
   - **Lack of Baselines for Comparative Insight**: To better evaluate the performance of continual learning methods, the paper should include non-continual learning baselines, such as pooling all data into a single learning process, training individual models per task, and then ensembling their performance.
   - **Insufficient Comparative Value in Forgetting Reduction**: The results only indicate that HFT reduces forgetting compared to FFT, which is not a surprising outcome, as HFT involves modifying fewer parameters during training. Other PEFT methods would likely yield similar results.
   - **Parameter Selection Strategy in Table 4**: The choice of selection strategies in Table 4 lacks intuitive appeal. Selecting 50% of parameters at the parameter level, rather than matrix-level, layer-level, or category-level, would be more straightforward and would precisely control the selection ratio.
4. **Questionable Ease of Implementation Argument**  
   - The paper claims that "HFT does not change the model architecture, making it easy to implement and scale," but this does not set HFT apart from other PEFT methods like LoRA, which are already available in easy-to-use APIs and require minimal code for implementation and scaling.

[1] Mitigating Training Imbalance in LLM Fine-Tuning via Selective Parameter Merging.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: See weaknesses.
soundness: 2
strengths: (1) The proposed method is simple and effective.

(2) The forgetting problem is an important problem and the authors offers a solution that tackles this problem.
summary: This paper proposes half finetuning as a method to solve catastrophic forgetting problem for large language models. HFT freezes half of the parameters and updating the other half. The experimental results demonstrate the effectiveness of HFT.
weaknesses: (1) The proposed method is too simple, too heuristic without novelties. Although the authors attempt to provide some theoretical explanations on page four, they are not convincing.

(2) Why HFT outperforms FFT? Because the number of trainable parameters of HFT is less than FFT, I think FFT could be deemed as an upper bound. In my understanding, the experiments are a trade-off of balancing downstream tasks and pre-training tasks. However, for many SFT-ed checkpoints, people would care more about the downstream abilities. 

(3) This paper proposes a method to mitigate forgetting. In Section 4.2, the authors compare the method with some forgetting mitigation methods but there is one important and simple baseline missing: the model average, which is a strong baseline and should be included. Could you explain why model averaging was not included in the comparisons? It would be beneficial if you could incorporate this baseline into their experimental setup to provide a more comprehensive evaluation.