Balancing Speciality and Versatility: a Coarse to Fine Framework for Supervised Fine-tuning Large Language Model (2024)

Abstract: Aligned Large Language Models (LLMs) showcase remarkable versatility, capable of handling diverse real-world tasks. Meanwhile, aligned LLMs are also expected to exhibit speciality, excelling in specific applications. However, fine-tuning with extra data, a common practice to gain speciality, often leads to catastrophic forgetting (CF) of previously acquired versatility, hindering the model’s performance across diverse tasks. In response to this challenge, we propose CoFiTune, a coarse to fine framework in an attempt to strike the balance between speciality and versatility. At the coarse-grained level, an empirical tree-search algorithm is utilized to pinpoint and update specific modules that are crucial for speciality, while keeping other parameters frozen; at the fine-grained level, a soft-masking mechanism regulates the update to the LLMs, mitigating the CF issue without compromising speciality. In an overall evaluation of both speciality and versatility, CoFiTune consistently outperforms baseline methods across diverse tasks and model scales. When compared to the full-parameter SFT, CoFiTune offers an average versatility improvement of 14%, while only incurring a marginal loss in speciality. Lastly, based on further analysis, we provide a speculative insight into the information forwarding process in LLMs, which helps explain the effectiveness of the proposed method. The code is available at https: //github.com/rattlesnakey/CoFiTune.

heading: 1 Introduction
text: Aligned LLMs mainly undergo a two-step procedure: initial pre-training on web-scale text corpora, followed by fine-tuning on diverse instructions to align with human intentions. They exhibit remarkable versatility, showcasing their ability to handle
†Work was done during internship at SenseTime Research * Corresponding author $ Project lead
various real-world tasks, such as reasoning, common sense question-answering, and instruction following (Zhao et al., 2023; Achiam et al., 2023; Lu et al., 2023b).
Despite the versatility, aligned LLMs still fall short in certain tasks or domains, such as mathematics (Gou et al., 2023), finance (Li et al., 2023c), and law (Cui et al., 2023a). To bolster performance in these particular tasks or domains, i.e., to gain speciality, a common practice is fine-tuning. However, during the fine-tuning process, the modification of model parameters often leads to catastrophic forgetting (CF), thereby causing a noticeable loss of versatility (Lin et al., 2023b). This loss adversely affects the performance of fine-tuned models across various real-world tasks (Cheng et al., 2023; Dong et al., 2023), propelling several works to investigate and contribute solutions to the CF in LLM versatility (Lin et al., 2023b; Wang et al., 2023c).
As a relatively new problem, the CF in LLMs remains under-explored. We categorize the existing studies into regularization-based, weight-based, and architecture-based methods1. Regularizationbased methods (Lin et al., 2023b; Smith et al., 2023) add extra terms into loss function to penalize parameter changes. Weight-based methods (Wortsman et al., 2022; Ke et al., 2022) introduce weight coefficients for parameters to regulate their updates. Architecture-based methods (Wang et al., 2023d; Razdaibiedina et al., 2023) design and exclusively
1See Appendix A.5 for more details about these methods.
7467
fine-tune extra modules outside the original model. However, these methods still have limitations: 1) Regularization- and weight-based methods involve fine-tuning all parameters, which poses a significant challenge to preventing CF in versatility. Particullary, CF deteriorates as the training iteration proceeds (Luo et al., 2023b). 2) Architecturebased methods only update the inserted parameters, which blocks the learning of speciality, particularly on challenging tasks (Lin et al., 2023b).
These limitations raise a key question: How can the model gain speciality while preserving the versatility, thereby enhancing its overall abilities? (shown in Fig. 1). Recent research threads highlight two pivotal findings: the presence of redundant parameters in the model (Bansal et al., 2023; Sun et al., 2023) and the distinct role played by different modules and layers (Geva et al., 2021; Meng et al., 2022a). These suggest that it is highly feasible to enhance speciality by updating specific parameters within a defined layer range while keeping the remainder frozen to maintain versatility.2
Therefore, we propose a Coarse to Fine framework, i.e., CoFiTune. At the coarse-grained level, we perform an empirical tree-search algorithm to identify and update specific modules within a defined layer range that are crucial for speciality without significant versatility penalty. Simultaneously, the remaining parameters are frozen to further preserve versatility. Subsequently, recognizing that not all units3 in a module are evenly important for the versatility (Michel et al., 2019), we utilize a fine-grained soft-masking mechanism to regulate the backward gradient flow based on their importance values for versatility. This further mitigates the CF issue while not jeopardizing the speciality. To summarize, our contributions are as follows:
1) We present CoFiTune, a framework striking a delicate balance between versatility and speciality. We also lead in creating a comprehensive Chinese CF setting, contributing to CF research in the Chinese domain. Extensive experiments demonstrate the effectiveness of CoFiTune across diverse tasks and model scales.
2) Our CoFiTune achieves over 95% versatility and 90% speciality compared to original and full SFT models on average, and we find that only tuning the FFN module in the mid-to-lower layer
2The results in Sec. 4.3 and 4.4, along with the insights 1, 3, and 4, further support our speculation.
3Units denote the attn. heads in MHA or neurons in FFN.
range achieves satisfactory speciality without significantly affecting versatility.
3) We conduct extra experiments to give more insights, demonstrating the module’s importance for gaining speciality and exploring the function of three crucial areas within the model.

heading: 2 Related Work
text: CF in LLM The previous studies about the CF problem typically focused on the CF of prior training tasks in a continual learning context, where a sequence of tasks are learned (Madotto et al., 2021; Wu et al., 2021). Recently, attention has shifted towards investigating the CF problem in the general abilities of LLMs. Luo et al. (2023b) first explored the CF problem in LLMs, revealing a pronounced CF phenomenon in general knowledge and reasoning. Subsequently, Wang et al. (2023c) established a benchmark comprising a sequence of challenging tasks to facilitate CF research in LLMs. Different from them, Lin et al. (2023b) proposed several solutions to address the trade-off between versatility and speciality in LLMs under a single task finetuning setting. In this study, we follow Lin et al. (2023b) to strike a balance between the speciality and versatility of LLMs.
Key Components in Transformer Previous works demonstrate the presence of substantial redundant parameters in Transformer-based models (An et al., 2020; Xia et al., 2022; Bansal et al., 2023). Therefore, it’s crucial to pinpoint the key components and precisely comprehend their underlying mechanisms. Recent endeavors try to analyze the involved layers and modules. Elhage et al. (2021) developed mathematical tools to unveil an information-copying behavior within the attention module. Geva et al. (2021) analyzed the feed-forward network (FFN), considering the up and down projection matrices as the key and value of the memories. Mirzadeh et al. (2024) demonstrated that SwiGLU, a common trainable activation function in the FFN of LLMs, can be replaced with non-trainable ReLU to reduce computation without affecting performance. In light of the aforementioned insights, in this work, we explore the key components, i.e., attention and FFN4 modules in certain layer range that are integral to gaining speciality and preserving versatility.
4For simplicity, starting from this section, FFN only refers to the combination of up & down projection.

heading: 3 The Framework
text: We start with the task formulation (Sec. 3.1) and the backbone we used in this work (Sec. 3.2). Then, we outline our CoFiTune framework, covering: 1) an empirical tree-search algorithm to identify modules within a defined layer range that balances speciality and versatility effectively (Sec. 3.3); 2) a FineSoftMask mechanism regulating the backward gradient flow based on units’ importance for versatility to further mitigate the CF issue (Sec. 3.4).

heading: 3.1 Task Formulation
text: Different from earlier researches, which focus on the CF in previously learned abilities when finetuning on a sequence of tasks (Liu et al., 2021b; Zhang et al., 2022b), we align with Lin et al. (2023b), emphasizing the trade-off between speciality and versatility in fine-tuning original LLM for a single task. This is because the original versatility of LLM is crucial and undergoes a progressive reduction at each fine-tuning step (Luo et al., 2023b). Additionally, we investigate a potential strategy to replace sequential fine-tuning for better performance. Further discussion of our setting is in Appendix A.4.
During supervised fine-tuning (SFT), given an input token sequence x = (x0, x1, . . .), the model is trained to predict the next token xi in an autoregressive manner:
LSFT(θ) = Ex∼DSFT
[ − ∑
i
log p(xi|x<i; θ) ] (1)
where θ represents the parameters of aligned LLM, DSFT is the fine-tuning dataset.
After obtaining the fine-tuned model θ̂, we assess its speciality (Spec.) and versatility (Vers.) scores. A unified (Uni.) score is further defined to evaluate its overall ability, i.e., both Spec. and Vers.. Refer to Sec. 4.2 for more details of scores.

heading: 3.2 Backbone Architecture
text: In this work, we utilize the Llama-based (Touvron et al., 2023) aligned LLM as our backbone, which consists of an Embedding module, an LM Head, and a stack of Llama Layers. Each Llama Layer integrates a multi-head attention (MHA), followed by a feed-forward network (FFN) with residual connections. For an input x, the Llama Layer generates output y based on the following equations:
x′ = x+ MHA(Norm(x))
y = x′ + FFN(Norm(x′)) (2)
Each head in the MHA module can be defined as:
headi = Attention(xW Q i ,xW K i ,xW V i ) (3)
where WQi , W K i , and W V i are the weight matrices of query, key, and value for the i-th head. The FFN module is parameterized by an up projection (up_proj) and a down projection (down_proj):
FFN(x′) = σ(x′W1)W2 (4)
where σ is the SwiGLU activation function (Shazeer, 2020), W1 and W2 are up_proj and down_proj respectively. More details of backbone information are illustrated in Appendix A.1.

heading: 3.3 Coarse-grained Level
text: Following the analysis in paragraph 2 of Sec. 2, which reveals the presence of parameter redundancy and key components (MHA and FFN) in LLMs, we aim to pinpoint the module (e.g., FFN) within a defined layer range (e.g., 10th - 20th layers) target for speciality without significantly affecting versatility. Subsequently, we exclusively train the identified module within the layer range,
while freezing the remaining parameters to further preserve the versatility.
Our exploration process encompasses three steps: 1) identifying the optimal broad layer range lr_broad; 2) narrowing down from lr_broad to obtain the optimal narrow layer range lr_narrow; 3) exploring the best modules within the narrow layer range lr_narrow. The best solution is up to the best Uni. score through three steps. Note that in steps 1 and 2, all modules, including both MHA and FFN are trainable in the specific exploring layer range; in step 3, only the selective module is trainable.5
Take the exploration process for Finance task in the 13B model as an example (illustrated in
5We exclusively train the up_proj W1 and down_proj W2 within the FFN (shown in Eq. 4) and head’s query WQi , key WKi , value W V i in the MHA (shown in Eq. 3).
6The details of tasks and models are illustrated in Sec. 4.1.
tain the best module “(10, 20] − FFN”. To delve deeper, we further examine the Uni. scores of submodules (up_proj and down_proj) within the best module (FFN). Here, we opt not to explore smaller layer ranges within lr_narrow (e.g., (10, 15] and (15, 20]) due to a substantial speciality gap compared to the all-layer range. See Appendix A.2 for the results of smaller layer ranges.
We select the best Uni. score among three steps as the final solution of the coarse-grained level. The entire exploration is illustrated in Algorithm 1.

heading: 3.4 Fine-grained Level
text: Recognizing that not all units (attention heads or neurons) in a module contribute equally to the LLM versatility (Michel et al., 2019), 1) we first compute their importance for LLM versatility; 2) Subsequently, we utilize a fine-grained soft-masking (Fine-SoftMask) mechanism to control the backward gradient flow based on their importance values, aiming to further mitigate the CF issue.
In this section, we use “module” or m to denote the module from the final solution identified at the coarse-grained level. Note that the Fine-SoftMask is exclusively applied to these modules, e.g., the FFN modules within the layer range (10, 20].
Computing Importance of Units Before directly fine-tuning the module, we follow Ke et al. (2022) to use a proxy based on robustness, i.e., KL-divergence loss, to compute units’ importance for versatility without accessing the external data. This is achieved by employing a gradient-based importance detection method (Michel et al., 2019):
ôm = gm ⊙ om
Im = 1
K
K∑
k=1
∣∣∣∣ ∂LImpt ∂gm ∣∣∣∣ (5)
LImpt = KL(f1LLM(x), f2LLM(x)) where om refers to the output of module m. K is the data size of DSFT. gm, an all 1’s virtual parameter, remains unchanged during the computing process, as we only need its gradient ∇gm (the term within || in Eq.5) on each parameter to get the importance of corresponding unit. ⊙ denotes element-wise multiplication, i.e., each gm,i in gm corresponding to a unit in the module. The overall importance of units Im is computed by averaging the gradient values of gm, where Im is of the same size as gm. f1LLM and f 2 LLM are the LLM with different dropout masks7. By simply feeding the same x ∈ DSFT to the model with different dropout masks twice, we obtain two representations. The difference between these two representations, computed by the KL-divergence loss, can measure the unit’s importance for the model. The rationale is as follows: if an unimportant neuron is dropped out, the resulting representation will show minimal alteration. Therefore, we will obtain a "normal type" representation similar to the one without any dropout. Conversely, if an important neuron is dropped out, the resulting representation will significantly change, resulting in an "abnormal type" representation (since dropout simulates noise addition). The difference between abnormal and normal output representations will be considerable. By utilizing KL divergence loss, we can effectively capture this difference and assign high gradient values to determine the importance of neurons. In contrast, the difference between normal representations will be marginal, resulting in small gradient values and, consequently, small importance values.
Fine-SoftMask Mechanism Then, we regulate the backward gradient flow based on the computed Im. We initially derive the original gradient ∇m by employing LSFT loss defined in Eq. 1. Subsequently, we apply the Im to obtain the modified gradient ∇̂m for updating:
∇̂m = (1− Im)⊙∇m (6) 7As the Llama layer lacks dropout module, we manually add it after the output of down_proj and the attention weights of MHA when computing the importance of units.
Here, we expand (by copying) the Im to match the dimensions of∇m to apply it to all associated parameters. This mechanism further mitigates the CF issue by regulating the update intensity of parameters based on Im for versatility. Note that the Fine-SoftMask is only applied in the backward pass during fine-tuning and the virtual parameter gm mentioned in the computing stage will be discarded in this process. The overview of our CoFiTune is shown in Fig. 2.

heading: 4 Experiment
text: In this section, we conduct extensive experiments to answer the following questions: 1) RQ1: Training which layers and modules can optimize the trade-off between speciality and versatility? 2) RQ2: How does our CoFiTune framework compared to the baseline methods? 3) RQ3: Can the Fine-SoftMask mechanism further alleviate the CF issue in versatility? 4) RQ4: Can our CoFiTune be seamlessly applied to models across different model scales and families?
We also conduct further experimental analysis to delve into more insights for this field in Sec. 4.4.

heading: 4.1 Datasets and Experiment Settings
text: Datasets: Building upon our task formulation in Sec. 3.1, we establish a Chinese CF setting, aiming to advance research in the Chinese language. This fills a crucial gap in existing studies, which have predominantly focused on English (Razdaibiedina et al., 2023; Lin et al., 2023b; Luo et al., 2023b).
Our setting emphasizes both speciality and versatility. Specifically, for LLM’s speciality, we select tasks with considerable complexity in two primary categories: 1) Knowledge Intensive Tasks: These tasks require the model to generate responses to questions tailored for specific domain. We create the Finance and Law instruction format datasets based on FiQA (Wang et al., 2023b) and LlamaLawyer (Huang et al., 2023); 2) Ability Intensive Tasks: These tasks aim to test LLM’s ability to handle challenging and crucial tasks in real-world scenarios. We create the mathematical problem solving (Math) and Context-Aware Generation (CAG8) based on MGSM8k (Chen et al., 2023a) and DuReader (He et al., 2018).
To evaluate the versatility of LLM, we combine the insights from previous studies (Lin et al., 2023b;
8CAG task requires the model to generate answers flexibly based on the given context (See Sec. A.6.2 for more details).
Luo et al., 2023b) and focus on three key aspects: 1) General Domain Knowledge (Gen-Kn.): We adopt CMMLU (Li et al., 2023b) to assess the original world knowledge stored in LLM; 2) Generic Reasoning (Gen-Rs.): We utilize commonly used reasoning datasets, including LogiQA (Liu et al., 2021a), LogiQA2 (Liu et al., 2023), OCNLI (Hu et al., 2020), and Zh-Winograd (Muennighoff et al., 2022); 3) Instruction Following (Instruct.): Following Lin et al. (2023b), we employ a Chinese GPT4-instruct dataset (Peng et al., 2023). Both Gen-Kn. and Gen-Rs. are evaluated in a zero-shot manner. Statistics and detailed descriptions of each dataset are provided in Appendix A.6.
Experimental Setting: We use the aligned LLM Chinese-Alpaca-Pro (Cui et al., 2023b) due to its robust versatility across multiple model scales, ranging from 7B to 33B, as detailed in Table 9. Implementation details, including learning rates and batch sizes tailored to different model scales and tasks, can be found in Appendix A.7. Fine-tuning adheres to the instruction prompt provided by Cui et al. (2023b), detailed in Appendix A.9.1.

heading: 4.2 Baselines and Metrics
text: Baselines: We compared our CoFiTune with full parameter SFT (Full SFT) and five CF baselines, with detailed descriptions provided in Appendix A.5. These baselines are carefully categorized into three groups:
• Regularization-based These methods introduce additional terms into the loss function to constrain changes in model weights. Selected baselines include L1 (Panigrahi et al., 2023) and L2 regularization (Lin et al., 2023b);
• Weight-based These methods design weight coefficients for parameters to control their updates. Selected baselines include WiseFT (Wortsman et al., 2021) and V-SoftMask (Ke et al., 2022);
• Architecture-based These methods fine-tune only the external module, leaving the rest of the parameters fixed. Selected baseline includes LoRA (Hu et al., 2021).
Metrics: The exploration algorithm in Sec. 3.3 requires numerous experiments, demanding a fast, cost-effective, and accurate automatic evaluation strategy. For Finance, Law, and CAG, we employ automatic generation metrics encompassing both
semantic alignment and n-gram matching, including BERTScore (Zhang et al., 2020), Rouge (Lin, 2004), and BLEU (Papineni et al., 2002). Additional experiments confirm a strong correlation between the automatic scores and those generated by GPT-4, as well as human annotations. Moreover, a new evaluation strategy is introduced to improve the reliability of the evaluation results. The details of the correlation evaluation and the new evaluation strategy are provided in Appendix A.18. For Math, we employ a rule-based extraction (Chen et al., 2023a) method to obtain accuracy. Utilizing the lm-evaluation-harness framework (Gao et al., 2021), we assess the accuracy of datasets in general domain knowledge (Gen-Kn.) and generic reasoning (Gen-Rs.). The instruction following (Instruct.) is evaluated using log-likelihood (LL) following the approach of Lin et al. (2023b). To evaluate overall performance in both speciality (Spec.) and versatility (Vers.), we define Spec. = 1 3(BERTScore+BLEU+Rouge) in Finance, Law and CAG, while Spec. = accuracy in Math. We define Vers. = 13(Gen-Kn. + Gen-Rs. + Instruct.) and Uni. = Spec. + Vers., (Uni. ∈ [0, 2]). More details of evaluation metrics are available in Appendix A.10.

heading: 4.3 Results
text: Optimal Layer Range and Module (RQ1) The exploration algorithm mentioned in Sec. 3.3 is carried out on four tasks under 7B and 13B models. Due to space limits, we present overall results for the Finance task under the 13B model in Fig. 3 as an example, with detailed results available in Appendix A.12 for future analysis in this field.
As depicted in Fig. 3, during step 1, the optimal Uni. score is attained with the “(10, 30] - MHA & FFN” configuration. Notably, the Spec. scores in the bottom and middle layer ranges (e.g., (0, 20], (5, 25], and (10,30]) are relatively high and comparable, whereas the top layer range (e.g., (20, 40]) exhibits the lowest performance. Conversely, the Vers. score is lowest in the bottom layer range and improves as the layer range ascends. Similar findings can also be observed in the 7B model. Further analysis of this observation is provided in Sec. 4.4.
In step 2, the best Uni. score is achieved with “(10, 20] - MHA & FFN”, where a reduction in the number of trainable layers from 20 to 10 results in an increase in Vers. score and a decrease in Spec. score. However, the decline in Spec. score
is less pronounced than the increase in Vers. score, summing up to a higher Uni. score in overall performance. Moving to step 3, the Uni. score for “(10, 20] - FFN” surpasses all other configurations. Specially, the Spec. score of “(10, 20] - FFN” is comparable to “(10, 20] - MHA & FFN” while its Vers. score is superior. Furthermore, the Spec. score of FFN significantly outperforms other modules including MHA, down_proj, and up_proj. The module importance for speciality will be further examined in Section 4.4.
Back to the whole picture, upon examining the results across various tasks and model scales, we surprisingly observe a consistent pattern:
Insight 1. The “(N× 25%,N× 50%] - FFN” configuration yields the best Uni. score on all tasks for both 7B and 13B models.
Namely, the optimal Uni. score of the coarsegrained level search for all tasks under the 7B model (i.e., N = 32) is achieved with “(8, 16] - FFN”, while for the 13B model (i.e., N = 40), it is “(10, 20] - FFN”. Moreover, for certain special scenarios, we offer an optional solution in Appendix A.13.
Performance of CoFiTune (RQ2) We summarize the Uni. scores of our CoFiTune and the com-
CoFiTune CoFiTune
w/o Fine-SoftMask Spec. Vers. Spec. Vers.
Finance 0.4351 0.4945 0.4352 0.4912 Law 0.4503 0.4903 0.4512 0.4855 Math 0.1120 0.5130 0.1080 0.5102 CAG 0.5406 0.5105 0.5397 0.5054
Law tasks compared to not applying it, and even a slight improvement in CAG and Math tasks. Moreover, Fine-SoftMask contributes to a Vers. score improvement of 0.7%, 1%, and 1.1% in Finance, Law and CAG tasks respectively. Similar conclusions in the 7B model are presented in Table 10.
Performance of CoFiTune across Models of Different Scales and Families (RQ4) To verify the generalization of CoFiTune on different model scales and families, we directly apply Insight 1, obtained in RQ1, without tree-search exploration to the Chinese-Alpaca-Pro 33B model and another popular aligned model9, BLOOMZ (Muennighoff et al., 2023). The overall results of the 33B model are shown in Table 4, with detailed results in Appendix A.15.3. For the BLOOMZ model, we conduct experiments on 3B and 7B scales under Finance and Math tasks, the overall results are presented in Table 5, with detailed results in Appendix A.14. As indicated in Table 4, CoFiTune maintains its superior unified performance under the 33B model, exhibiting improvements in Uni. score by 1.2%, 1.3%, and 1.6% in Finance, CAG, and Math tasks compared to the best-performing baseline Wise-FT. Similarly, Table 5 demonstrates the strong competence of our CoFiTune under the BLOOM family model, showcasing improvements
9In this paper, unless explicitly specified, the model referred to is Chinese-Alpaca-Pro.
in Uni. score by 5.9% and 6.4% in the Finance task when compared to Full SFT baseline. These results further validate the widespread effectiveness of Insight 1 and CoFiTune across models of different scales and families.10

heading: 4.4 Further Analysis
text: Finance Math 0.4
0.42
0.44
0.46
0.48
0.5
Fin an
ce S
pe c.
MHA & FFN MHA FFN down_proj up_proj
0.04
0.06
0.08
0.1
0.12
0.14
0.16
M at
h Sp
ec .
layer ranges in both 7B and 13B models. Note that there are two non-overlapping layer ranges, (0, N × 25%] and (N × 50%, N × 75%], situated between (0, N × 50%] and (N × 25%, N × 75%]. However, the Vers. score of (N × 50%, N × 75%] is favorable (shown in Fig. 3 and Appendix A.12); therefore, we suspect that it is the (0, N × 25%] that harms LLM’s versatility.
Thus, we conduct experiments in this range with detailed Vers. score of Math task under 13B model in Fig. 6. In Fig. 6, we observe that: 1) The detailed Vers. scores of “(0, N × 25%] - MHA & FFN” significantly lag behind “(N × 25%, N × 50%] - MHA & FFN”; 2) In the layer range (0, N × 25%], compared with “FFN & MHA”, FFN further impairs versatility while MHA mitigates it. Similar findings across various tasks and model scales are shown in Appendix A.17. Hence, we derive the following insight:
Insight 4. The LLM’s versatility may predominantly reside in the layer range (0, N × 25%], particularly within the FFN module.
Furthermore, combining the hints from Geva et al. (2021), Meng et al. (2022b), Allen-Zhu and Li (2023b), Allen-Zhu and Li (2023a), and Chuang et al. (2023) with our findings in this work, we speculate the forward process in Fig. 5: 1) MHA mod-
ules in Versatility Area aggregate information, enriching input for corresponding FFN. FFN modules utilize this enriched input information to extract relevant knowledge, passing it through residual connections to Optimal Speciality Gaining Area; 2) Optimal Speciality Gaining Area handles both versatility and speciality information, forwarding it to Augmenting Area. 3) Processed information from the previous two areas is forwarded to Augmenting Area through residual connections, focusing on information enhancement for final prediction.
In this context, modifying modules in Versatility Area impairs the original memory and abilities of LLM, impeding the flow of versatility information passing through. In contrast, updating modules in Optimal Speciality Gaining Area allows versatility information to be recalled and passed through residual connections. Extra experiments are scheduled to validate these speculations in the future.

heading: 5 Conclusion
text: In this work, we strive to optimize the balance between LLM’s speciality and versatility. Our proposed CoFiTune framework employs an empirical tree-search exploration algorithm to identify the module within a defined layer range that is crucial for gaining speciality without significantly affecting versatility. Additionally, the Fine-SoftMask mechanism is applied to further alleviate CF issue without impairing the speciality. We introduce a Chinese CF setting to advance the research in Chinese domain. The experimental results demonstrate that our CoFiTune outperforms all baselines across various tasks and model scales. Our in-depth analysis provides valuable insights for future work.

heading: 6 Limitations
text: In this study, we opt not to use the rehearsal-based method, which involves replaying a small portion of the general dataset during fine-tuning (Rolnick
et al., 2019; de Masson D’Autume et al., 2019). This decision stems from the uncertainty surrounding the replay data ratio and strategy in this domain. Increasing the number of replay samples during fine-tuning would also incur higher training resource costs. Moreover, we believe that the rehearsal-based method can be integrated with our proposed CoFiTune framework, and we plan to explore this in future work.
Additionally, the model employed in this study is mainly based on the Llama architecture. This choice is driven by the fact that Llama family models demonstrate remarkable dominance and performance in the current NLP research area (constituting over 90% on the LLM leaderboard12). Besides, based on the fact that most current LLMs share the same architecture as Llama, i.e., transformer decoder-only architecture, we are confident that our identified universal optimal range can also be applied to a majority of models within the same architecture (the experiment results of BLOOMZ in RQ4 have validated this point). Therefore, our CoFiTune framework can be directly adopted in these models using Insight 1, without incurring any tree-search exploration overhead. Furthermore, we plan to explore other model architectures (e.g., encoder-decoder) in future research.

heading: A Appendix

heading: A.1 Detailed Information of Llama Backbone
text: The MHA operation is a crucial component of the transformer, defined as:
MHA(Q,K,V) = [head1; . . . ; headh]WO (7)
where ; is the concatenation operation, Q, K, and V are the query, key, and value matrices, respectively, and WO is a learnable output matrix unique to the MHA module. Each head is computed as:
headi = Attention(xW Q i ,xW K i ,xW V i )
Attention(Qi,Ki,Vi) = Softmax ( QiK T i√
dk
) Vi
(8)
where WQi , W K i , and W V i are the corresponding weight matrices of query, key and value for the i-th head. The FFN module is parameterized by an up projection matrix (up_proj), followed by a down projection matrix (down_proj):
FFN(x′) = ( SwiGLU(x′)⊗ (x′W1) ) W2 SwiGLU(x′) = SiLU(x′W) RMSNorm(x′) = w ⊙ x′√
Var(x′) + ϵ
(9)
where x′ is the input tensor of FFN, W is the learnable weight matrix without bias (a.k.a gated projection matrix), W1 and W2 are up_proj and down_project respectively. SiLU(x′) = x′ ⊗ Sigmoid(x′). ⊗ is the element-wise multiplication operation. w represents the learnable weight parameter, Var(x′) denotes the variance of x′ across the last dimension, and ϵ is a small constant introduced for numerical stability.
As the Llama layer lacks dropout module, we manually add it after the output of down_proj and the attention weights of MHA when computing the importance of units in Sec. 3.4.
Given Mirzadeh et al. (2024)’s demonstration that SwiGLU, a commonly used trainable activation function in the FFN of LLMs, can be substituted with non-trainable ReLU to decrease computation without compromising performance, we have chosen not to investigate SwiGLU in our exploration outlined in Sec. 3.3. Specifically, our exploration focuses solely on the up_proj and down_proj components within the FFN.

heading: A.2 Smaller Exploring Layer Range Results
text: There is a substantial speciality gap between the smaller layer range and the all-layer range. For example, in Math under the 7B model, the smaller layer range (N × 25%, N × 37.5% accounts for only 37.5% of (0, N ] in Accuracy.

heading: A.3 Exploration Algorithm in Coarse-grained Level
text: The exploration at coarse level is illustrated in Algorithm 1. The variables results1, results2, results3 are dictionary data structures intended to store results from steps 1, 2, and 3, respectively. The variables left, right, pivot represent the start index of their corresponding layer range. α denotes the number of layers targeted for coverage. For instance, in a 13B model (N = 40), α = 20 in step 1 and α = 10 in step 2. The GetBestLayers function aims to identify the layer range that performs best in terms of the unified score. The GetSubModules function returns the sub-modules of a specific module; for example, the sub-modules of FFN are up_proj and down_proj. The functions GetMax and GetSecond are responsible for obtaining the start index of the layer range that achieves the best and second-best performance. f∗_layer_range denotes the parameters of FFN and MHA in the ∗_layer_range that are trainable in the model, while others are frozen. Similarly, fm∗_layer_range represents the parameters of a specific module in the ∗_layer_range that are trainable in the model, while others are frozen. The Max function returns the optimal strategy that achieves the highest unified score among all three steps.
Algorithm 1 Exploration Algorithm in Coarse Level 1: Input: Llama Model f(·), N layers, evaluation data D 2: Initialize: results1, results2, results3 ← {}, left← 0, right← N × 50%, α← N × 50% 3: Function:
def GetBestLayerRange(left, right, α, result): 4: pivot← (left+ right)//2 5: left_layer_range← (left, left+ α] 6: right_layer_range← (right, right+ α] 7: pivot_layer_range← (pivot, pivot+ α] 8: result[left]← Eval(fleft_layer_range(·), D) 9: result[right]← Eval(fright_layer_range(·), D)
10: result[pivot]← Eval(fpivot_layer_range(·), D) 11: left← GetMax(result) 12: right← GetSecond(result) 13: pivot2 ← (left+ right)//2 14: pivot2_layer_range← (pivot2, pivot2 + α] 15: result[pivot2]← Eval(fpivot2_layer_range(·), D) 16: return GetMax(result)
# Step 1: 17: lr_broad_start_idx← GetBestLayerRange(left, right, α, results1)
# Step 2: 18: left← lr_broad_start_idx; right← lr_broad_start_idx+ α; α← α//2 19: lr_narrow_start_idx← GetBestLayerRange(left, right, α, results2)
# Step 3: 20: lr_narrow ← (lr_narrow_start_idx, lr_narrow_start_idx+ α] 21: for m← [MHA,FFN] do 22: results3[m]← Eval(fmlr_narrow(·), D) 23: end for 24: best_module←Max(results3) 25: for msub ← GetSubModules(best_module) do 26: results3[msub]← Eval(fmsublr_narrow(·), D) 27: end for 28: return Max(results1, results2, results3)

heading: A.4 Discussion of our setting
text: Recall the conventional settings in continual learning, where a pre-trained model fθ0 undergoes sequential fine-tuning on tasks [T1, T2, ..., TK], and forgetting evaluation is based on the model’s performance on Ti after being trained on Tj (i < j).
In contrast, our investigation focuses solely on forgetting in the versatility of the original model during the fine-tuning of a single task. This setting is motivated by two key considerations:
• Aligned LLMs have demonstrated remarkable versatility, achieved through extensive pre-training on a massive dataset comprising tens of billions to even hundreds of billions of samples. This enables them to effectively handle a wide range of real-world tasks (Chowdhery et al., 2023; Achiam et al., 2023). However, adopting a sequential fine-tuning method poses a significant risk of progressively undermining the model’s versatility at each fine-tuning step (Luo et al., 2023b; Wang et al., 2023c). Additionally, the severe decline in versatility not only inhibits the fine-tuned performance of the model on diverse tasks (Cheng et al., 2023; Dong et al., 2023) but also hinders the effectiveness of transferring knowledge from it when gaining speciality (Jie et al., 2022; Luo et al., 2023a).
• Recent studies suggest an alternative to the traditional sequential fine-tuning approach. Instead of fine-tuning tasks sequentially, each task is independently fine-tuned on a pre-trained model, and their weights are subsequently combined through interpolation (Ilharco et al., 2022; Li et al., 2022; Liu and Soatto, 2023; Yu et al., 2023). This weight interpolation approach significantly enhances performance and mitigates catastrophic forgetting compared to conventional methods that sequentially fine-tune one model on [T 1, T 2, T 3...].

heading: A.5 Baseline Descriptions
text: In this Section, we describe the baseline method in our setting in detail. We carefully categorize them into three classes:

heading: A.5.1 Regularization-based Methods
text: L1 Regularization Panigrahi et al. (2023) introduced explicit L1 regularization (with a strength of 0.001) on the parameter shift θ → θ̂, denoted as |θ̂ − θ|. This regularization strategy helps alleviate catastrophic forgetting by limiting the extent of tuning.
L2 Regularization Building upon the concept introduced by Xuhong et al. (2018), Lin et al. (2023b) utilized a more consistent parameter regularization approach. Specifically, the divergence between the parameters of the fine-tuned model θ̂ and the pre-trained model θ serves as the object of the L2 penalty in the optimization process, denoted as |θ̂ − θ|22. This approach helps prevent deviation from the pre-trained model during tuning.

heading: A.5.2 Weight-based Methods
text: Wise-FT The Wise-FT (Wortsman et al., 2021) methodology involves two distinct stages: initially finetuning the pre-trained model θ on a specific downstream task to obtain the fine-tuned model θ̂, followed by the fusion of the original pre-trained model θ and the fine-tuned model θ̂ using linear weight interpolation f(1−α)θ+αθ̂, where α is a hyper-parameter ranging from 0 to 1. Parameter Constraint-Based Methods formulate their approach around parameter-specific strategies, leveraging either parameter efficiency techniques or the estimation of parameter importance.
Vanilla Soft-masking Ke et al. (2022) proposed Vanilla Soft-masking to address CF in continual domain pre-training of language models. Specifically, a gradient-based detection method is used to compute the importance value of units within the attention and FFN modules across all transformer layers for general domain knowledge. The resulting importance vector is then employed to control the backward gradient flow. Importantly, the soft-masking regulation is exclusively applied to the backward process, ensuring that knowledge transfer across domains during tuning remains unaffected.

heading: A.5.3 Architecture-based Methods
text: LoRA Grounded in the assumption that the alteration in weights during model adaptation exhibits a low "intrinsic rank," Hu et al. (2021) introduced a low-rank matrix into the dense layers within the network. This allows the indirect training of these layers by optimizing the decomposition matrices within the low-rank. Throughout training, all the decomposition matrices within the low-rank matrix are trainable, while the pre-trained weights remain frozen to preserve the original general ability of the pre-trained model.

heading: A.6 Statistics and Descriptions of Datasets
text: Currently, LLM still falls short in certain areas (Li et al., 2023a; Lu et al., 2023a), we mainly select two types of tasks to evaluate its speciality. The overall statistics of the selected task datasets are shown in Table 7, and the detailed descriptions are as follows:

heading: A.6.1 Knowledge Intensive Tasks
text: • Finance: To create a Chinese financial QA instruction format dataset, we adopt the building procedure from Chen et al. (2023b). Initially, we translate the instruction content of the English FiQA13 dataset into Chinese. Subsequently, we employ the translated Chinese instructions to generate responses that align with China’s national conditions using a robust Chinese FinLM14 (Chen et al., 2023b).
• Law: Huang et al. (2023) initially gather Chinese legal questions from OpenLaw (Chen, 2018) and JEC-QA (Zhong et al., 2020). They then employ ChatGPT to obtain responses, constructing the LlamaLawyer15 dataset. This dataset comprises both single-turn and multi-turn instructions. For a fair comparison, we exclusively choose single-turn instructions in our setting.

heading: A.6.2 Ability Intensive Tasks
text: • Math: Chen et al. (2023a) establishes a multi-lingual GSM8k dataset (MGSM8k)16 by translating the original English GSM8k (Cobbe et al., 2021) dataset into ten different languages using ChatGPT. In our setting, we specifically select the Chinese version within MGSM8k for both training and evaluation.
13https://huggingface.co/datasets/FinGPT/fingpt-fiqa_qa 14https://huggingface.co/Go4miii/DISC-FinLLM 15https://github.com/AndrewZhe/lawyer-llama 16https://mathoctopus.github.io/
• Context-Aware Generation (CAG17): To enhance the model’s capability of leveraging the context information to generate answers (Tong et al., 2023, 2024), we follow the methodology RA-DIT mentioned in Lin et al. (2023a) to construct an instruction format dataset based on Dureader (He et al., 2018), a Chinese Reading Comprehension (RC) dataset that includes three types of questions: “Entity”, “Description” and “YesNo”. In our setting, we specifically choose “Description” type questions (Dureader-Desc), because their answers are human-written, presenting a significant difference from the context content. Following the methodology of Chen et al. (2023b), we design a prompt (illustrated in Appendix A.9.2) that allows the model to flexibly generate appropriate responses under contexts with different properties. We select the context that is irrelevant to the corresponding question to form a negative sample. This aims to enable the LLM to ignore misleading context content and lean into its parametric knowledge to respond. In contrast, we form positive samples by selecting context relevant to the corresponding question, intending to evaluate whether the LLM can better utilize relevant background knowledge to make a prediction. During the evaluation of the Context-Aware Generation (CAG) task, we use a subset of negative samples from the training data, paraphrase their instructions during inference, and utilize automatic generation metrics to assess the model’s ability to leverage its parametric knowledge for response. We use positive samples outside the training data to evaluate the model’s proficiency in utilizing context for response.

heading: A.6.3 General Domain knowledge
text: We employ CMMLU (Li et al., 2023b) for a zero-shot evaluation of the model’s general domain knowledge. CMMLU18 covers a wide range of subjects, comprising 67 topics that span from elementary to advanced professional levels. It includes subjects that require computational expertise, such as physics and chemistry, as well as disciplines within humanities and social sciences. Notably, many tasks within CMMLU involve contextual nuances and wording that may not easily translate across languages. Moreover, numerous tasks have answers specific to China, making them contextually relevant but potentially not universally applicable or considered correct in other regions or languages.

heading: A.6.4 Generic Reasoning
text: We assess the generic reasoning ability of LLM through the following datasets in a zero-shot manner:
• LogiQA19: The dataset follows a paragraph-question pair format, each accompanied by four candidate answers. It is sourced from publicly available logical examination papers for reading comprehension, designed by domain experts to assess participants’ logical reasoning ability. Therefore, the questions exhibit reliable quality and cover a diverse range of topics.
• LogiQA220: This is the second version of the LogiQA dataset, collected from the Chinese Civil Service Entrance Examination. The dataset includes newly released exam questions and practice questions, sourced from approximately 20 provinces in China where the exam is held annually. Exam materials are made publicly available on the Internet after each exam, and practice questions are obtained from various sources.
• OCNLI21: The Original Chinese Natural Language Inference dataset (OCNLI) is the first extensive Natural Language Inference (NLI) dataset for Chinese. Unlike other datasets, OCNLI does not depend on automatic translation or non-expert annotation. Instead, it gathers annotations from native speakers with expertise in linguistics.
• Zh-Winograd: XWinograd (Emelin and Sennrich, 2021)22 is a multilingual Winograd dataset 17The CAG task essentially serves as the generation step in Retrieval-Augmented Generation (RAG), flexibly relying on the
retrieved context to generate responses. 18https://huggingface.co/datasets/haonan-li/cmmlu 19https://github.com/lgw863/LogiQA-dataset 20https://github.com/csitfun/LogiQA2.0 21https://github.com/CLUEbenchmark/OCNLI/ 22https://huggingface.co/datasets/Muennighoff/xwinograd
designed for assessing commonsense reasoning and coreference resolution. In our setting, we exclusively choose the Chinese version for evaluation.
A.6.5 Instruction Following Dataset Chinese GPT4-Instruct encompasses a variety of user-oriented instructions, spanning areas such as email writing, social media, and entertainment. This dataset was introduced by Peng et al. (2023), who rigorously followed the methodology outlined by Taori et al. (2023) to implement the self-instruct strategy (Wang et al., 2022) with GPT-4.
A.7 Implementation Details
We use DeepSpeed Zero3 (Rajbhandari et al., 2021) and Adam optimizer (Kingma and Ba, 2014) to train all the models. Our experiments are conducted on a computing platform equipped with 32 V100 GPUs. Each experiment is run with three different random seeds, and the results are averaged to obtain the final outcome. The maximum sequence length is set to 512 across all four tasks, and we employ a greedy decoding strategy for generating results. A cosine scheduler with a 3% warm-up period is applied. In our study, LoRA is applied to five weight matrices: Wq, Wk, Wv in the MHA module, and Wdown, Wup in the FFN module. Following Lin et al. (2023b), we experiment with lora_rank values in [4, 8, 16], setting lora_alpha = 2× lora_rank to achieve optimal performance. For Wise-FT, consistent with Lin et al. (2023b), we explore α values in [0.4, 0.6, 0.8] to determine the best performance. Different learning rates and global batch sizes are set for training tasks across various model sizes, with detailed information provided in Table 8.

heading: A.8 Details of Different Model Scales
text: All the model adheres to the Apache-2.0 license, the details of different model scales are illustrated in Table 9.

heading: A.9 Prompts
text: A.9.1 Instruction Prompt
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction: {instruction} {input}
### Response: {output}
We use “{instruction}”, “{input}”, and “{output}” to replace the specific instruction, input, and output.
A.9.2 CAG Prompt
请根据参考材料回答下面的问题。下面的材料可供参考。 “Please answer the following questions based on the reference materials. The following materials are available for reference.” （注意：1、材料可能与问题无关，请忽略无关材料，并基于已有知识回答问 题。2、尽量不要直接复制材料内容作为答案，而应该将材料内容作为事件的补充 与潜在分析，启发思考）； “(Note: 1. The material may have nothing to do with the question. Please ignore the irrelevant material and answer the question based on existing knowledge. 2. Try not to directly copy the material content as an answer, but use the material content as a supplement and potential analysis of the event to inspire thinking. );” 参考材料: “references:” {reference} 问题: “question:” {question}
Here, the contents labeled with “” contain the corresponding English translations of Chinese. We use “{reference}” and “{question}” to replace the specific reference and question.

heading: A.10 Evaluation Metrics
text: A.10.1 Instruction Following Score Following LMflow (Diao et al., 2023), we assess instruction following performance using log-likelihood:
NLL =− 1 N
N∑
i=1
log p(sentencei|contexti)
=− 1 N
N∑
i=1
log p(tokeni,1, tokeni,2, ..., tokeni,ni |contexti),
where ni is the length of the token in sentencei. As the LL results are typically in the order of several hundreds (e.g., 531.27), and the results of Accuracy or automatic generation metrics are values ranging from 0 to 1, they inherently possess disparate scales. To comprehensively assess both the speciality and versatility of LLM, we initially scale the LL by dividing it by one thousand. Since a lower LL is indicative of superior performance, we then subtract the result from 1 to obtain the final instruction following (Instruct.) score, i.e., Instruct. = 1− LL/1000.

heading: A.10.2 Speciality Score
text: For Finance, Law, and CAG tasks, we follow Tan et al. (2020, 2021); Zhang et al. (2022a); Kong et al. (2022b,a); Zhang et al. (2023) to utilize the automatic metrics to evaluate the quality of generated text ,
i.e., BERTScore23, BLEU24, and Rouge25 to evaluate their speciality performance. The Rouge metric comprises Rouge-1, Rouge-2, and Rouge-L. We initially average the results of Rouge-1, Rouge-2, and Rouge-L to obtain the overall Rouge result. Subsequently, we further average the results of BERTScore, BLEU, and Rouge to derive the final speciality score. It’s worth noting that we employ the F1 score in both Rouge and BERTScore.
Rouge = Rouge-1 + Rouge-2 + Rouge-L
3
Spec. = BERTScore + Rouge + BLEU
3 (10)
For Math, as it only has one metric, i.e., Accuracy, we directly use its Accuracy result as the Spec. score.
A.10.3 Versatility Score We assess the versatility of LLM in three aspects: general domain knowledge (Gen-Kn.), generic reasoning (Gen-Rs.), and instruction following (Instruct.). For evaluating the generic reasoning of LLM, we average the results from four datasets to obtain an overall generic reasoning (Gen-Rs.) score. Subsequently, we average the scores of Gen-Kn., Gen-Rs., and Instruct. (computed in Appendix A.10.1) to derive the overall versatility (Vers.) score for LLM.
Gen-Rs. = LogiQA + LogiQA2 + OCNLI + Zh-Winograd
4
Vers. = Gen-Kn. + Gen-Rs. + Instruct.
3 (11)

heading: A.10.4 Unified Score
text: To facilitate the evaluation of LLM in both speciality and versatility, we sum the results of the speciality (Spec.) score (obtained in Appendix A.10.2) and the versatility (Vers.) score (obtained in Appendix A.10.3) to get the final unified (Uni.) score:
Uni. = Spec. + Vers. (12)
23https://github.com/Tiiiger/bert_score 24https://www.nltk.org/api/nltk.translate.bleu_score.html 25https://github.com/pltrdy/rouge

heading: A.11 Ablation Results of Fine-SoftMask

heading: A.12 Exploration Results in Coarse Level
text: A.12.1 7B Results
A.12.2 13B Results

heading: A.13 Optional Solution
text: In our context, instruction following is a crucial aspect of LLM’s versatility. However, in scenarios where users can provide clear and concise instructions, the model’s precision in this aspect may be less critical. We conduct additional analysis on the results of LLM’s versatility without instruction following (Vers. w/o instruct), considering different trainable modules in the layer range (N × 25%, N × 50%] under 7B and 13B models. The derived Uni. score without instruction following (Uni. w/o instruct) shows that “(N × 25%, N × 50%] - MHA & FFN” generally outperforms “(N × 25%, N × 50%] - FFN”. This suggests that “(N × 25%, N × 50%] - MHA & FFN” could be a viable alternative when users value instruction following aspect less.
A.14 Detailed Results of CoFiTune and Baselines under BLOOMZ Backbone
A.15 Detailed Results of CoFiTune and Baselines under Chinese-Alpaca-Pro Backbone A.15.1 7B Results
A.15.2 13B Results
A.15.3 33B Results

heading: A.17 Exploring CF in LLM’s Versatility

heading: A.16 Module Importance for Speciality

heading: A.18 Details of Our Evaluation Setting

heading: A.18.1 A New Strategy for Improving the Reliability of Evaluation Results
text: The exploration algorithm mentioned in Sec. 3.3 requires extensive experiments conducted to explore the distinct role of each layer range and modules within it. This necessitates a fast, cost-effective, and accurate automatic evaluation strategy. For Finance, Law, and CAG, we employ automatic generation metrics encompassing both semantic alignment and n-gram matching, including BERTScore (Zhang et al., 2020), Rouge (Lin, 2004), and BLEU (Papineni et al., 2002). The automatic results for Finance under the 7B model in the unseen test set are presented in Table 54. As depicted in Table 54, our observations indicate that the evaluation results across different methods for the unseen test set are similar and generally demonstrate suboptimal performance. This finding is not unique, as previous studies (Ovadia et al., 2023; Wang et al., 2023a) in question-answering have encountered similar issues26, and diverse retrieval-augmented generation (RAG) approaches (Ovadia et al., 2023; Gao et al., 2023) are proposed to enhance LLMs’ performance in addressing this issue. Another possible reason is the inadequacy of current automatic evaluation metrics in gauging LLMs’ output for unseen and dynamic questions (Chang et al., 2023; Wang et al., 2023a).
Consequently, to enhance the reliability of the automatic evaluation results, we advocate for a novel evaluation approach when evaluating Finance, Law, and CAG27 tasks: initially sampling instructions from the training set and subsequently paraphrasing them to form the test questions, i.e., sample-then-paraphrase strategy. Intuitively, this strategy facilitates valid automatic evaluation of LLMs’ generation by testing in-domain knowledge while addressing concerns related to data contamination through test instruction rephrasing (Schaeffer, 2023; Oren et al., 2023).
A.18.2 Validation of the Automatic Results To further validate the effectiveness of the automatic metrics used in assessing the quality of LLMs’ generation, we randomly sample a set of outputs from the financial CoFiTune 7B model and evaluate their Spec. score with both GPT-4 and human annotators.
For human evaluation, we randomly select 100 samples and ask a qualified annotator28 to assess each output’s quality based on aspects such as accuracy, relevance, and coherence. Each score can range from 1 (very poor) to 5 (very good). For GPT-4 evaluation, we adhere to the same criteria and create a prompt instructing GPT-4 to assign a rating to each model output. An example of our evaluation prompt for GPT-4 is presented in Table 55.
Upon acquiring individual scores from both the annotator and GPT-4, we normalize each score to a range between 0 and 1. Subsequently, we employ the Inter Annotator Agreement (IAA) metric to assess
26Fine-tuned models become static data snapshots during training and may swiftly become inadequate for effectively supporting dynamic scenarios, making them less capable of handling unseen domain-specific questions.
27The test set of CAG task comprises both positive and negative samples, and their evaluation methods slightly differ; for more details, refer to the CAG task description in Appendix A.6.2.
28The annotator has scored above 600 on the College English Test 6 level (CET-6) and is compensated with 5 RMB per sample.
the concordance between our automatic Spec. scores and the scores obtained from both the annotator and GPT-4. As illustrated in Table 56, our Spec. score reveals a substantial level of agreement with evaluations conducted by both human annotators and GPT-4. This alignment underscores the efficacy and robustness of the Spec. score as a reliable metric for assessing the quality of outputs generated by LLMs in our evaluation setting (mentioned in Appendix A.18.1).

heading: Future Work
text: Moreover, we believe that the rehearsal-based method can be integrated with our proposed CoFiTune framework, and we plan to explore this in future work. Additionally, the model employed in this study is mainly based on the Llama architecture. This choice is driven by the fact that Llama family models demonstrate remarkable dominance and performance in the current NLP research area (constituting over 90% on the LLM leaderboard12). Besides, based on the fact that most current LLMs share the same architecture as Llama, i. e. , transformer decoder-only architecture, we are confident that our identified universal optimal range can also be applied to a majority of models within the same architecture (the experiment results of BLOOMZ in RQ4 have validated this point). Therefore, our CoFiTune framework can be directly adopted in these models using Insight 1, without incurring any tree-search exploration overhead. Furthermore, we plan to explore other model architectures (e. g. , encoder-decoder) in future research