Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Authors: Yifan HAO
Xingyuan Pan
Hanning Zhang
Chenlu Ye
Rui Pan
Tong Zhang
Venue: ICML 2025 poster (2025)
Keywords: model ensemble
fine-tuning
multi-task
Primary area: theory->domain_adaptation_and_transfer_learning
TL;DR: This paper mainly focuses on understanding the benefits of ensemble in supervised fine-tuning.

Abstract: Supervised fine-tuning (SFT) on domain-specific data is the dominant approach for adapting foundation models to specialized tasks. However, it has been observed that SFT models tend to forget knowledge acquired during pretraining. In vision models, ensembling a pretrained model with its fine-tuned counterpart has been shown to mitigate this issue. In this work, we demonstrate that the same holds for language models, and, more strikingly, we observe an overadaptation phenomenon: the ensemble model not only retains general knowledge from the foundation model but also outperforms the fine-tuned model even on the fine-tuning domain itself.
Despite the empirical success of ensembling, a theoretical understanding of its benefits remains underexplored. We develop a formal theoretical analysis of the overadaptation phenomenon.
 Ensembling mitigates this by balancing two primary sources of error: bias, caused by insufficient fine-tuning, and variance, introduced by overfitting to fine-tuning data. While regularization techniques aim to address this trade-off, we show that ensembling provides a more effective solution. We analyze this phenomenon in over-parameterized linear settings and demonstrate that interpolating between pretrained and fine-tuned weights significantly improves performance. These findings offer theoretical justification for the observed advantages of model ensembling, supported by empirical experiments consistent with our analysis.

Understanding Overadaptation in Supervised Fine- -ine-Tuning: The Role of Ensemble Methods
Yifan Hao
Xingyuan Pan
Hanning Zhang" 1 Chenlu Ye!
Rui Pan Tong Zhang'
Abstract
1. Introduction
With the remarkable success of large language models (LLMs) such as GPT-4 (Achiam et al., 2023). Gemini (Team et al., 2023), and Claude (Anthropic, 2023), the pretrain- finetune paradigm has gained significant attention for its outstanding performance. Supervised fine-tuning (SFT) is a widely adopted approach for adapting foundation models to specific downstream tasks. However, a well-known chal- lenge with SFT models is the tendency to forget information acquired during pre-training (McCloskey & Cohen, 1989; Goodfellow et al., 2013). Model ensembling, also known as model averaging, has emerged one of the most effective strategies to address this issue. Its benefits have been em- pirically demonstrated in vision models (Wortsman et al., 2022b) and in reinforcement learning from human feed- back (RLHF) (Lin et al., 2024). By simply interpolating the weights of pre-trained and fine-tuned models, ensembling has shown competitive performance in mitigating forgetting compared to other approaches. In this work, observe that the same advantage extends to supervised fine-tuning of LLMs. Moreover, beyond its effectiveness in mitigating forgetting on upstream tasks, we also observe a surprising phenomenon of overadaptation, which reveals that model ensembling can outperform the fine-tuned model even on downstream tasks, where the fine-tuned model expected excel, which also aligns with previous results in Wortsman et al. (2022b); Lin et al. (2024).
Supervised fine-tuning (SFT) on domain-specific data is the dominant approach for adapting foun- dation models to specialized tasks. However, it has been observed that SFT models tend to forget knowledge acquired during pretraining. In vision models, ensembling a pretrained model with its fine-tuned counterpart has been shown to mitigate this issue (Wortsman et al., 2022b). In this work, we demonstrate that the same holds for language models, and, more strikingly, adaptation phenomenon: the ensemble model not only retains general knowledge from the founda- tion model but also outperforms the fine-tuned model even on the fine-tuning domain itself. De- spite the empirical success of ensembling, the- oretical understanding of benefits remains un- derexplored. We develop formal theoretical analysis of the overadaptation phenomenon. En- sembling mitigates this by balancing two primary sources of еттor: bias, caused by insufficient fine- tuning, and variance, introduced by overfitting to fine-tuning data. While regularization techniques aim to address this trade-off, we show that en- sembling provides more effective solution. We analyze this phenomenon in over- parameterizeed linear settings and demonstrate that interpolating between pretrained and fine-tuned weights sig- nificantly improves performance. These findings offer theoretical justification for the observed ad- vantages of model ensembling, supported by em- pirical experiments consistent with our analysis.
observe over-
However, despite the impressive empirical effectiveness of ensemble methods, the corresponding theoretical insights are still limited, especially in the context of modern over- parameterized model. Most theoretical studies on ensem- bling have focused on traditional mnder-parameterized set- tings (Brown al., 2005; Lin et al., 2023; 2024), which do not align with the current use of over parameterizzed large neural networks. While some works (Allen-Zii -hu & Li, 2020; Hao et al., 2024) have demonstrated the benefits of en- sembling independently trained models, these only address improvements in disttribbuiin (OOD) robustness. To the best of our knowledge, existing work has addressed the central question:
'Equal contribution 'University of Illinois Urbana-Chann Champaign, Illnoos. Correspondence to: Tong Zhang <tongzhang @tongzhang- ml.org>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
Why does ensembling achieve such remarkable efficiency, enhancing both generalization on downstream tasks and

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
mitigating forgetting on upstream tasks?
fitting by achieving better balance in the "bias- variance" trade-off, thereby reducing test error;
In this work, we address this question by investigating the role of ensembling in addressing the overadaptation phe-
4. (Ensemble for improved trade-off management): Com- bining the pre-trained model with either the ridge or "ridgeless" fine-tuned model enables more effective balance of the "bias-variance" trade-off in test error, leading to improved performance on Task 2.
nomenon. During supervised fine-tuning, specialized fine- tuned models overly focus on downstream tasks, leading to loss of valuable information retained in the pre-trained model. The effectiveness of ensembling can be attributed to its ability mitigate overadaptation. By simply averaging the weights of the pre-trained and fine-tuned models, the ensemble recovers the information lost during fine-tuning while preserving the knowledge gained from the fine-tuning process.
And for the forgetting phenomenon, we consider the perfor- mances on both Task and Task establishing that
1. (Impact of regularization in forgetting): Without hurt- ing the performance on Task 2, ridge regression mit- igates forgetting on Task 1 by balancing the "bias- variance" trade-off, effectively managing both pre- trained and fine-tuned errors simultaneously.
Specifically, we start with presenting empirical evidence in Section 3 that highlights the harmful effects of overadapta- tion and demonstrates the efficiency benefits of ensembling in both improving fine-tuning performance and mitigating forgetting. Building on these observations, we develop formal theoretical framework in Section 4 and Section 5 to analyze the effectiveness of model ensembling, focusing on its impact on fine-tuning tasks and pre-traine pre-training task reten- tion. We attribute the improved performance of ensembling to its ability better balance the "bias-variance" trade- off in test error. To simplify our explanation, we model the pre-training and fine-tuning processes using an over- parameterized linear setup. Within the context of canonical linear regression, we represent the pre-trained model as the "ridgeless" estimator on Task For fine- seeettnnigg on Task 2, total overfitting (overadaptation) is characterized by "ridge- less" regression, while non-overii approaches, such as early stopping, are captured through ridge regression (Lin & Rosasco, 2017; Lu et al., 2022). Our main theoretical findings can be summarized as follows.
2. (Ensemble for enhanced forgetting mitigation): Lever- aging the pre-trained and fine-tuned models, ensem- bling further improves such "bias-variance" balance, providing an additional reduction in forgetting.
1.2. Empirical validation overview
Our theoretical analysis is mainly inspired by the "magical" empirical results, which are deferred to Section 3. Here we highlight the main results, to show the consistency be- tween our theoretical results and the empirical phenomenon. Specifically, give empirical evidences showing that:
1. (Harmful overadaptation in fine-tuning): When the training process extends into the "overfitting" regime (without applying early stopping), fine-tuning perfor- mance deteriorates;
1.1. High-level theoretical insights
2. (Enhanced performance through ensemble); Model ensemble consistently improves model performance across various experiment settings, both achieving bet- ter performance on fine-tuning tasks and mitigating forgetting pre-training tasks efficiently.
On the theoretical side, based on an over- parameterized regression setup, after pre-training on Task 1, we fine-tume tune the model on a specific Task 2. The results are stated on two aspects as follows.
Focusing on the performance on Task 2, we prove that
2. Related Works
1. (Poor Performance of pre-trained Model): The pre- trained model exhibits high "bias" term in test error due to its inability to capture task-specific information for Task 2;
There exists a substantial body of work on model ensemble, pre-training and fine-tuning. In this section, we review the most relevant works to ours.
2. (Limitations of fine-tuning without regularizer): Fine- tuning without any regularization, i.c., using a "ridge- less" estimator, results in a high "variance" term in test error due to overfitting (overadaptation) on noisy data, leading to poor performance;
Model ensemble. Model ensemble has been a popular technique to enhances generalization performance, docu- mented in the existing literature (Hansen & Salamon, 1990; Krogh & Vedelsby, 1994; Perrone & Cooper, 1995; Opitz & Maclin, 1999; Dietterich, 2000; Zhou et al., 2002; Polikar, 2006; Rokach, 2010; Rame et al., 2022; Arpit et al., 2022;
3. (Impact of regularization in fine-tuning): Applying ridge regression during fine-tuning helps mitigate over-
Kumar et al., 2022a; Wortsman et al., 2022a; Lin al.,
2

---

Understanding Overadaptation in Supervised Fine-Tuning, The Role of Ensemble Methods
2023). Recently, ensemble the pre-trained and fine-tuned models has been verified to benefit the out-of-distribution robustness (Wortsman et al., 2022b), as well as decreasing forgetting in reinforcement learning from human feedback (Lin et al., 2024).
5.0
Regularized Fine-tuning
Vanilla Fine-tuning
5.6
MT-Bench Scores
5.4
Understanding model ensemble. On the theoretical side, works explaining the good performance of ensemble are limited, especially in the context of overparameterized mod- els. In traditional underparameterrzed settings, Brown et al. (2005) decomposes the prediction error of ensemble models into bias, variance and a covariance term between individual models, proposing algorithms to encourage the diversity of individual models to reduce the covariance term; similarly, Lin et al. (2023) showed that ensembling two independently trained models increases feature diversity, thereby improv- ing ofiistirbbutiin (OOD) robustness, while Lin et al. (2024) extended this framework to demonstrate that fea- ture diversity also mitigates forgetting on upstream tasks. In the of wver-parameterized models, recent works such as Allen-Zhu & Li (2020) and Hao et al. (2024) have shown that ensembling independently trained models im- proves OOD robustness. A comprehensive explanation for the improved model efficiency remains lacking.
52-
5.0
Training Epochs
Figure I. Early Stop Experiments: The performance on MT- Bench when the training epech increases. We conduct vanilla fine-tuning and DiffNorm--enalty fine-tuning with Llama-3-8B on Dolly dataset.
3.1. Datasets and Benchmarks
Our experiments utilize the Dolly dataset (Conover et al., 2023), popular instruction-following dataset that covers a wide range of tasks, including Creative Writing, Closed QA, Open QA, Summarization, Information Extraction, Classifi- cation and Brainstorming, ensuring its high-diversity.
Pre-training and fine-tuning, The pre-training and fine- tuning paradigm have become a comerstone in developing high-performance models across various domains, partic- ularly in large language models (LLMs)). Recent advance- ments have focused on enhancing this framework through various techniques. The LP-FT technique, introduced by Kumar et al. (2022b), involves initializing the pre-trained feature extractor with a reasonably good classifier; Huang et al. (2021) proposed low-rank adaptation (LoRA) to re- duce the number of trainable parameters during fine-tuning, which benefits parameter-effictee training: Tian al. (2023) presented a trainable projected gradient method aimed at enhancing out-of-distribution (OOD) generalization; model ensemble has also demonstrated effectiveness in improv- ing performance, as evidenced by many studies (Cha et al., 2021; Chu et al., 2022; Wortsman et al., 2022a; Lin et al., 2024).
The LLMs' instruction-folloingg ability is evaluated on MT-Bench (Zheng et al., 2023) with single-answer grad- ing. This benchmark prompts conversational assistants with challenging multi-turn open-ended questions and utilizes "LL -aaju--- for evaluation, which comprises 80 ques- tions, evenly distributed across 8 categories: Writing, Role- play, Extraction, Reasoning, Math, Coding, Knowledge I (STEM), and Knowledge II (humanities/ssoiall science). For each response, the LLM judge of GPT-4 will provide a score on a scale of to 10, indicating the overall instruction- following ability of the evaluated conversational assistant.
We also assess LLMs' general ability on MMLU (Hendrycks et al., 2021) and Commonsense-QA (Talmor et al., 2019). MMLU is dataset containing 57 tasks in- cluding mathematics, chemistry, computer science, law, and more, which measures multi-task ability and requires extensive world knowledge and problem-solving ability. Commonsense- QA contains more than 10K real-world com- mon sense questions. It requires LLMs to identify related roal-world knowledge and distinguish the distracted an- swers.
3. Motivated by the Intriguing Phenomenon
We start our observation on instruction-folloiing fine- tuning tasks, highlighting the empirical findings from three key aspects: (i) the impact of using a regularizer versus overfitting (overadaptation)), the performance of ensembling pre- pre-trained and fine-tuned models on (ii) fine-tuning tasks and (iii) pre-training tasks.
The goal of the experiments is to highlight the strengths of regularization and ensembling in improving the tradeoff between instruction following and general abilities, reveal-
3

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
ing that these common generalization techniques not only enhance downstream task performance, but also alleviate forgetting issues in LLMs.
Overfitting (overadaptation) is harmful during the fine- tuning process. In Figure 1, we show the training process the Dolly dataset and performance on MT-bench without applying early-stopping. It becomes evident that perfor- mance deteriorates quickly with additional epochs, indicat- ing harmful overfitting. Even with early stopping, as seen in Table 1, the use of regularizers, such as Korm-Penalty and DiffNorm- PeiNttt--enalty, improves generalization performance compared non-regularized fine-tuning (Vanilla-FT).
Performance Tradeoff
fine-tuee Nethods
Ensemble Methode
Ensemble Methods
75.25
Commnonsenss-OO QA
75.00-
74.75
AnSel
Ensemble enhances generalization performance. As shown in Table I, the ensemble of pre-trained and fine-tuned models consistently outperforms the individually fine-tuned models on fine-tuning tasks. This improved performance highlights the effectiveness of model ensembling. aligning with the empirical findings in Lin et al. (2024). Although both methods, Norm- Penalty and SiffNorm-Pennalee use early-stopping to prevent overfitting, they apply different penalties additionally in the training process. Interestingly, in our experiments, Norm-Penalty consistently outperforms DifffNorm- Penally in almost all settings, suggesting that the choice of regularizer plays a crucial role and warrants further exploration.
74.50
74.25
14.00
73.75
5.55
5.50
5.65
5.70
5.75
5.20
5.85
MT-Bencch
Figure 2. Experiments on Commonsense-OU QA and MT-bench: The performance tradeoff on MT-Bench and Commonsense-QU QA for ensemble methods and fine-tune methods. The ensemble meth- ods are based on the pre-trained model and the DiffNorm- DiffNorn-Penalty model, where the results of different r values and seeds are pre- sented here. We use Llama-3-8B in our experiments.
Performance Tradeoff
65.4
Fin-ture Methods
3.2. Experiment Settings
D+ET
Ensemble Wethods
Ensemblee Mathods
Our experiments are based on three well-known open-source base models: Llama-3-8B (Dubey et al., 2024), Qwen2-7B (Yang et al., 2024), and Gemma-2-9B (Team et al., 2024), where for each model, we compare the vanilla fine-tuning approach to variants with generalization techniques applied. Depending on whether or not ensembling is used, and which types of regularization are adopted, totaling 2 x 2 variants are proposed to be compared with the vanilla fine- tuning baseline.
1H03
64.5
THO
M4.
5.55
5.60
5.65
MT-Bench
For regularization, an additional penalty term of ||0 or ||0 - 112 is added the fine-tuning loss, where i means the fine-tuned model parameters and 0 represents the pre- trained weights. We denote those variants as Normal- Penalty and LiffNorm-Penale separately. For ensembling, the parameters of fine-tuned models are weighted-averaged with the pre-trained model. Those variants are denoted as Avg-Norm-Penall and Nyg-DiffNorm-Penatty respectively. We make our implementation publicly available 1. For more experimental details, please refer to Appendix A.
Figure.3 3. Experiments on MMLU and MT-hench: The perfor- mance tradeoff on MT-Bench and MMLU for ensemble methods and fine-tune methods. The ensemble methods are based on the pre- trained model and the DiffNorm fifieatt-Penalty model, where results of different r values and seeds are presented here. We use Llama- 3-8B in experiments.
Ensemble improves trade-off between pre-training and fine-tuning tasks. We also evaluate the trade-off between pre-training tasks (Commonsense--0) and MMLU) and the downstream task (MT-bench). The results are shown in Figure 2 and Figure 3. Compared to individually fine-tuned models, ensemble models achieve better trade-off's on these tasks especially when T is larger than 0.5. The ensem- ble models generally have a high MT-bench score while maintaining good performance on Commonsense-QA. and
3.3. Results
Our empirical results uncover three key issues in the stan- dard fine-tuning process, as shown in Figure 1, Table 1, Figure and Figure 3:
4

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
MMLU, The fine-tuned models would suffer from more forgetting if they achieve high MT-hench scores. The re- sults suggest that ensemble methods could achieve a good balance of instruction follooiinn ability and generalization ability.
The second objective function incorporates regularizer with parameter À avoid overfitting, in ridge regression:
[|i88-Y1] {iiIo --12 + Alle - 6all3]
arg min n
which implies a solution as
4. Problem Setup and Notations
0, X x -x0O)
(5)
Notation. For any matrix A, we use ||Al|2 to denote its & operator norm and use (rA to denote its trace. The i-th largest eigenvalue of A is denoted as (A) The transposed matrix of A is denoted as AA" And the inverse matrix of A is denoted as A The notation a = o(b) and a < mean that a/b - 0, a = (b) means that a/b + 0c, a = O(b) means that a/ /b is bounded, and = means a = (bb) as well as b- (a).
With the pre-trained estimator and fine-tuning estimator, we also consider the ensemble (weighted averaging) estimator:
0 =1 (1-111 - r)6 + T0, where
θ 0z2 6x
(6)
with an averaging coefficient 0 S S 1. In our settings, the performance measures for an estimator 0 are the excess mean squared errors on Task 1:
Given the significant performance improvements achieved through ensembling, we seek to understand its benefits in this section. To this end, we analyze its effects within the framework of over wver-parameterized linear regression.
Spre (0) v().. [<x 0 - (e.. [ )²) = =Ev, | - x=,0)
To be specific, the pre-training process is taken on Task 1, where n i.i.d. training examples (11, ..),...; (4ns Vn) from distribution D take values in R2 R. and obey the following linear model with parameter 0 E I2P:
and Task 2:
La (f) == (z.. [ 0 T..a - ( v. [ 3.)
Ez. [ x 0)).
Ely; x3] zi'o.
(1)
where the variables (æ., y.) and (ã., V.) are independent copies of (21, V1) and (ø 71) respectively.
To capture the strong performance of the pre-trained model, we consider the min-norm ("ridgeless") estimator on Task 1:
0 = x (xx""xx="-
(2)
Motivation for the objective function. The two objective functions used in the fine-tuning process serve to character- ize the presence or absence of early stopping. Specifically, if early stopping is not employed, the overadaptation on downstream tasks can be described as "ridgeless" regression. Conversely, if early stopping applied, it can be viewed as utilizing a ridge regularizer (Lin & Rosasco, 2017; Lu et al., 2022).
where X = [II,...,,.,] x,,,,, 1T E RTXP and Y [y1.... Vn.1T E R Starting with the pre-trained estima- tor dj, fine-tuning process is taken on Task 2, where n i.i.d. training examples (#1. ii.,..., (Tn, Fn) are sampled from another distribution D 2, as well as following another linear model:
- = T.'0.
(3)
In further analysis, we adopt the following assumptions on settings above:
During the fine-tuning process, we consider two scenarios. The first is "ridgeless" regression, defined by the following objective function:
1. On Task 1, x, = E2/2n, where Σ == E[ diag (Ai, ., Ap). and the components of f): are inde- pendent o₂ -ubbgausssi random variables with mean zero and unit variance;
arg min ||0 12, s.t. x0=
where
X
[ FI...., IJ E Rexp and Y Vnl T E Rd Accordingly, the estimator is as
2. On Task 2, I. = 51/27, where S == E = diag diag/A,..... Axll and the components of 77, are also independent a₂ og-subgaussian random variables with mean zero and unit variance;
0 0=:-0 0, + XT(XXT)
XO.).
(4)
FFor simplicity, we assume that the sample sizes for pre- training and fine-tuning are the same. However, our results remain valid even when using a larger dataset for pre-training compared to fine-tuning.
3. Ely; xf'0 x2, 0|2 E/c,]² = 02 > 0, E
x: 0 T. 0/2 =B 72 > 0:
5

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Table 1. MT-Bench of models fine-tuned on Dolly (Conover et al., 2023).
METHODS
REGULARIZER
ENSEMBLING
LLAMA-3-8B
QWEN2-7B
GEMMA-2-9B
VANILLA-FT
X
5.68
6.57
6.52
NORM-PENALTY |
X
5.84
6.81
6.59
DIFFNORM-PENALTY
|0 0:112
X
5.78
6.68
6.65
AYG-NORM-PENALI
|1013
/
5.96
7.10
6.83
AVG-DIFFNORM-PENA.. |10 0
5.85
6.84
6.89
4. The true model parameters 0, 0 are independent of samples, and could be decomposed as
5. Main Theorems and Interpretations
In this section, we present the test performance of various estimators and provide explanations for the improvement in both generalization fine-tuning tasks and forgetting mitigation on pre-training tasks achieved through model ensemble, highlighting the "brianvariance" trade-off phe- nomenon. To simplify our explanations, we consider the covariance matrices for x, I as follows,
0g + a1
0 0g + 022
where c,, 01 02 are independent with each other, as well as holding ||0.1 < oo, af I and E Elaza]] 224p. where Ç1, C2 0.
Discussion on the setting. Pre-trained models often achieve reasonably good generalization across various tasks, and fine-tuning serves to enhance their performance on some specifie tasks. Thus, it is reasonable assume that, despite the differences between Task and Task 2, they share many similarities, which makes the "benign overfitting" estima- tor 0, a good "initial point" in fine-tuning process. This leads us to posit that the true model parameters 0 and 0 share significant information, represented by Oc. while also exhibiting some differences characterized by ai and a2- Additionally, we assume that I and Y' share a large amount of same eigenvectors (see Condition 1), further reflecting the similarity between Task and Task
Condition 1. Denoting Σ = (i,.. and E:= diag ( (A,,... ...,, Ap), have
1, k,"
1, i 1= i=....,k"
k"
A
X
% k <is
y i> k.
0,
i> P.
And our main theorem is also based on the following condi- tion:
Condition 2. We consider the following three items:
(good performance of pre-trained model) For Task 1, there exists constant
The connection between theoretical and empirical re- sults. The theory aims to provide explanations for the benefits of ensemble, adopting a linear setting for intuitive insights. Such simplifications have been widely used in prior works (Mallinar et al., 2022; Kumar et al., 2022b). The connection between theoretical and empirical results be established by adopting NTK explanation, fine-tuning results in parameters close to pretraining points. Specifically, for nonlinear neural network fæz 8) in the NTK regime, we approximate it using a first-order Taylor expansion fxx 0) ~ f(x 0o) + Vof(æ, oo) (T 0n)- Comparing this with the linear setting = x= 0., we can interpret the "features" in neural networks, i.c.. fof(z, 1o). as the input the linear model, and the trainable parame- ter 0 bo the parameter 8, in y _TO Since f(x, v)) remains unchanged during training, its effect can be disre- garded in this simplification. And in our linear setup, the high-dimensional assumption on ø (see Condition 2) can characterize the "features" " Vof (x, 0o) in overparameter- ized neural networks.
such that
001), p p= wn)) p= o(n
2. (sparsity Task 2) For Task 2, the following conditions hold
B n. pxn, 7111, py 2c10" /C2
where c₁ 0 is constant only depending d.-
3. nog--nelle data noise and task difference) For the noise level and the "bias" parameter ar 02 we have
Ç = O(n -n,, .620 x5² Co = o(1),
C2 wmaxx(n (1-01/2 -))-
Discussion on the conditions. figh-dimensional eigenvalue structure characterized by sev- eral "large" eigenvalues alongside many "small" ones. This
Condition 1 describes a
6

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
structure aligns with the eigenvalue decay observed in gen- eral kernel matrices within deep neural networks (Fan & Wang, 2020; Li et al., 2024). We can fürther validate this assumption by analyzing the eigenvalue distribution of the Hessian matrix in practical models using PyHessian (Yao et al., 2020) and variants of Lanczos algoritbms(Zhang et al., 2024), which confirms the presence of several "large" eigen- values and many "small" eigenvalues. In Condition 2, the first item reflects the "benign" overfitting scenario discussed in Bartlett et al. (2019), which helps to characterize the good performance of the pre-trained model. The second item de- lineates the "sparse" structure of Task 2, i.e, there are many zero eigenvalues in Σ. Such sparse structure observed in fine-tuning tasks reflects the nature of knowledge special- ization across different inputs. While pretraining involves diverse inputs encompassing broad knowledge, fine-tuning is performed specific tasks with narrower scope, lead- ing to a "sparse" structure in our theoretical formulation. The third item outlines certain conditions that address the significance of data noise and the differences between the two tasks. Since the specific order relationships are techni- cal assumptions intended to clarify our theoretical results more clearly, we believe that relaxing them will not compro- mise our theoretical intuition. We consider this possible direction for further exploration.
The detailed proof is in Appendix C. The results highlight three key insights: (i) selecting an appropriate regularizer during fine-tuning helps reduce overadaptation on noisy samples, leading to improved generalization on fine-tuning task; ensembling the pre-trained and fine-tuned models can decrease overadaptation further, then (ii) enhances perfor- mance on fine-tuning task, as well as (iii) mitigating for- getting phenomenon on pre-training task. These benefits can be understood through a "bias-variance" trade-off phenomenon:
1. Both Spre and La contain "bias" term and "variance" term.
2. The pre-trained estimator 0 is mainly dominated by "bias" terms, as it is induced from a sufficiently high- dimensional distribution (see Condition 2). It performs poorly on Task 2 and achieves good performance on Task 1, because it only contains the information in pre- training process, and lacks information specific to Task 2, resulting in small "bias" term in Epee. as well as a large "bias" term in Lt.
3. On the other hand, the "ridgeless" estimator 8z though it minimizes "bias" error, overfits the noisy training data during fine- tuning causing a significant "variance" term in La and both large "bias" term and large "vari- ance" term in Lpre.
Before delving into main results in Theorem 5.1, we introduce two notations:
0²
X = nii
4. Introducing a proper regularizer has the ability to achieve better performance on La and Lpe + Ca. by balancing the "bias" and "variance" errors more effectively.
+(A) = (8X + nAI) -XX")
(FUI((X'' + +NM) -XXX"
+Gtrr(XXX "XX + + +
5.
The improved generalization on both La and Cpre + Las from model ensemble results from further balancing these error terms with a properly chosen weight т, which applies to both the "ridgeless" estimator f2 and the riggereegularized estimator 0x.
The main theorem is then stated as follows:
Theorem 5.1. For any 5 defined above, as Condition and Condition 2 are sutisfied, there exists constant c >>> 1 such that for ó E (0, 1) and In(1/8) < n8 with probabil- ity at least - ö over X,X,
Comparing with previous viewpoints. From traditional statistical perspective, which mainly focuses on limited model complexity. increasing the model complexity typ- ically results in a higher "variance" error and lower "bias" error (Zhang, 2023). This trade-off suggests that overfitting noisy training data leads to poor generalization and high test error. However, recent advancements have introduced the concept of "benign overfitting" (Bartlett et al., 2019). which suggests that sufficiently large models can achieve su- perior performance despite overfitting. In our analysis, the pre-training process mainly operates on a digh-diimnsional distribution D, facilitating strong performance on Task and aligning with the principles of "benign overfitting". Con- versely, the fine-tuning phase focuses on "sparse" structure,
1. (the effectiveness of regularization.)) for any 0 < S 2X, we have
(n( < La((s) Ca
2. (forgetting mitigation with model ensemble.) for any 0 < X < 2X and 7' 7'(A)/2 < 1, we have
(p < Lpc (o.) < (prc(02) La
3. (improving performance on ensemble.) for any &
X < and T'()) S T < 1, we have
(g) < Lu (0x).
7

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
i.e D, associated with limited model complexity. This lim- ited complexity explains the observed harmful overfitting during fine-tuning.
terms related to f2 (the difference between the pre-training and fine-tuning tasks), which can be denoted as the "bias" terms, and the terms related to ō (the variance of data noise in the fine-tuning task), which can be denoted as the "vari- ance" term. Insufficient fine-tuning leads to a large "bias" and small "variance", while overadaptation in fine-tuning results in large "variance" and small "bias". By effectively balancing this trade-off, model ensembling can achieve su- perior performance.
Empirical validation. We also conduct simulations across diverse settings to validate Theorem 5.1. The de- tails and results, summarized in Appendix B, demonstrate the strong performance of the ensemble model on both pre- training and fine-tuning tasks, aligning well with our theo- retical findings.
6.2. Estimator Performances Comparison
6. Proof Sketches of Theorem 5.1
The effectiveness of regular. After obtaining the results in Lemma 6.1, could compare the excess risk different estimators. The results
In this section, we summarize the proof sketch of Theo- rem 5.1, which mainly contains two steps, and the detailed proof is in Appendix C. For simplification, we take the following two notations in further analysis:
Le (02) < (f1)
could be induced directly from Condition 2. For the excess risk of ridge estimator, taking derivative with respect to A on its approximation
PRA = XXXX" (1XXX*+nAA-11 + Px = -
6.1. Excess Risks Approximation
Catr[(1-
+ x2 + +
First, we show that with a high probability, the excess risks corresponding to estimators in Equation (2), Equation (4), Equation (5) and Equation (6) could be expressed as: Lemma 6.1. As Condition 1 and Condition 2 are satisfied, there exist a constant >> 1 such that for E (0, 1) and /c, with probability at least 1 - over X, X, fo S X < /² (1(22) we have
we find that such excess risk will decrease with the increase of A within the range 0 X A T² (m(2)) which implies
(r)( < (02), 4A < X S 26 ((<6))
(8)
ln(1/8) < n$ n
Forgetting mitigation with model ensemble. The anal- ysis is similar. With the results in Lemma 6.1, (8x) + Lie (0x) is mainly dominated by
(,,) = (,)
(0( 2 (atr(11 - Px) +XX((
(atr[(1- - Px x + P2, E} + 2 tr + nA) -XXX
(a(()) = (atr((1- - (E)
+0²u((X8* +ndl) -XXX",
Taking derivative with respect to X, we find that such term will decrease while 0 < A < 27 / (n(2), which implies that (O)) + (r(( Crr (02) + La (02). w0 < X S 25
Ca(03) CatT((1- -55
+ tr +ndl)- -2 "X
and
/052/(1) (n(2).
(0)) < Cp2)
(pra(0r) < (02);
And considering the benefits of model ensemble, for any fixed À E (0, 25 ((((2)), ppee + (x( is mainly dominated by
Epce (0)) < Cpro (05);
Lpce (p2) ~ (2tr((XX') XXXX
+ +rrrr( "XXX"
- -tx((I-Txxx)) xxx,))
(0x) 2 ( D) + 72 (x + + -X
+ ( + ndI) -2 -XX
Lpro CoT tr Σト
+ 2 + +
Ft + -
While we take derivative with respect to T, such term will increase with the increasing of as 7' (A) < < 1. So we have
Proof. See details in Appendix C.2 and C.3.
(p)) + (m( < (Ō)) + Lr (Ōx).
Using Lemma 6.1, the terms La and Cpee + La in Theo- rem 5.1 can be primarily determined by two key factors: the
(10)
( 1.
8

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Improved fine-tuning performance nally, for any fixed À with range [), 22 (22((2)2), we could take derivative with respect to on the approximated excess risk of ensemble estimator:
on ensemble. Fi-
immediate societal impacts be emphasized in the context of this work.
References
(2tr((1-TPx,)) ( (xXX +nAl)
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint 0rX7v:2303.08774, 2023.
we found such excess risk will increase with the increasing o while ('()) S I 1, so we have
(g) < (((()),
(11)
Allen-Zhu, Z. and Li, Y. Towards understanding ensem- ble, knowledge distillation and self seff-distillation in deep learning, arXiv preprint arXiv:2012.098/1 2020.
while 0 S A /² (m(2) and r' (A) ST
Combing all of the results in Equation (7), Equation (8), Equation (9), Equation (10) and Equation (11), we could finish the proof of Theorem 5.1.
Anthropic, A. Introducing claude, 2023.
Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensemble of averages: Improving model selection and boosting per- formance in domain generalization. Advances in Neural Information Processing Systems, 35:8265-8277, 2022.
7. Conclusion and Discussion
In this work, we bridge the gap in understanding how ensem- bling pre-trained and fine-tuned models controls overadap- tation, as well as enhancing both downstream performance and mitigating forgetting on upstream tasks. Motivated by surprising empirical findings showing that ensembling not only improves tne-tuning outcomes but also preserves pre- trained knowledge, we provide a formal theoretical analysis within an over- ver-parameterized linear setting. Our results re- veal that ensembling mitigates overadaptation by effectively balancing the trade-off between "bias" and "variance" errors in excess risk-an issue that regularization alone may not fully resolve. This theoretical insight is further supported by experiments and simulations, which closely align with our predictions.
Bartlett, P. L., Long, M., Lugosi, G., and Tsigler, A.
Benign overfitting linear regression. arXiv preprint arXiv: 1990.1113000000 3130033, 2019.
Brown, G., Wyatt, J. L., Tino, P., and Bengio, Y. Managing diversity in regression ensembles. Journal of machine learning research, 6(9), 2005.
Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S., Lee, Y., and Park, S. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-22418,, 2021.
Chu, X., Jin, Y., Zhu, W., Wang, Y., Wang. X., Zhang, S., and Mei, H. Dna: Domain generalization with diversified neural averaging. In International conference on machine learning, pp. 4010-4034. PMLR, 2022.
Our results not only offer deeper theoretical understanding of ensembling in the context of pre-trained models but also provide practical guidance for enhancing the performance of fine-tuning strategies. This work lays foundation for future research into refining ensembling methods and exploring their application broader machine learning tasks.
Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free dolly: Introducing the world's first truly open instruction- tuned I1m, 2023. URL com. com.///lcg//2233//////////////// n- ww--oommerree www.omnerialll--ivalleeeee. htw.:aaabbeiisstuutttootttttt ed-11m.
Acknowledgements
This work was partially supported by NSF grant No. 2416897 and ONR grant No. N000142512318.
Dietterich, T. G. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pp. 1-15. Springer, 2000.
Impact Statement
This paper contributes foundational research in the areas of model ensemble within the machine learning community. Our primary goal is to advance the theoretical understanding for the efficient performance of ensemble methods. Given the scope of this research, we do not anticipate immediate ethical concerns or direct societal consequences. Therefore, we believe there are no specific ethical considerations or
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A.. Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret,

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
C.., Wu, C., Wong, C., Ferrer, C. C.,, Nikolaidis, C., Al- lonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olang D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy. E., Lobanova, E., Dinan, E., Smith, F. M., Radenovic, F., Zhang, F., Syn- naeve, G., Lee, G., Anderson, G. L., Nail, G., Mialon, G., Pang, G., Cucurell, G. Nguyen, H., Korevaar, H., Xu, H., Touvron, H.. Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I, Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong. J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J.. Spisak, J., Park, J., Rocca, J., Johnstun, J.. Saxe, J., Jia, J., Alwala, K. V., Upasani, K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K. Iyer, K., Malik, K., Chiu, K., Bhalla, K., Rantala-Yeary, L., van der Maaten, L.,, Chen, L.., Tan, L.. Jenkins, L., Martin, L., Madaan, L., Malo, L.,, Blecher, L., Landzaat, I.., de Oliveira, I.., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis. M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov. N., Bogoychev, N., Chatterji, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, P., Vasic, P., Weng. P., Bhargava, P., Dubal, P., Krishnan, P. Koura, P. S., Xu, P., He, Q., Dong, Q., Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabas- appa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S., Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang. S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S. Collot, S., Gururangan, S., Borodinsky, S., Her- man, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Petrovic, V., Chu, W., Xiong, W., Fu. W., Meers, W., Martinet, X., Wang, X., Tan, X. E., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y., Mao, Y., Coudert, 7. D., Yan, 7., Chen, 7., Papakipos, 7 Singh, A., Grattafiori, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Boesenberg, A., Vaughan, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Franco, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A., Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang, B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C. Wang. C., Kim, C., Zhou, C., Hu, C., Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Civin, D.,. Beaty, D., Kreymer, D., Li, D., Wyatt, D., Adkins, D., Xu, D.,
Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang, D., Le, D., Holland, D., Dowling, E., Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Brinkman, E., Arcaute, E., Dunbar, E., Smothers, E., Sun, F., Kreuk, F., Tian, F, Ozgenel, F,, Caggioni, F., Guzmán, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Thattai, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Shojanazeri, H., Zou, H., Wang. H., Zha, H., Habeeb, H., Rudolph, H., Suk, H., Aspegren, H., Goldman, H., Damlaj, I., Molybog, I., Tufanov, I., Veliche, I.-F., Gat, I. Weissman, J., Geboski, J., Kohli, J., Asher, J., Gaya, J.-B., Marcus, J.. Tang. J., Chan, J., Zhen, J.. Reizenstein, J., Teboul, J., Zhong, J., Jin, J., Yang. J., Cummings, J., Carvill, J.. Shepard, J., McPhie, J., Torres, J., Ginsburg. J., Wang. J., Wu, K., U, K. H., Saxena, K., Prasad, K.. Khandelwal, K.. Zand, K., Matosich, K., Veeraragha- van, K., Michelena, K., Li, K., Huang, K., Chawla, K., Lakhotia, K., Huang, K., Chen, L., Garg, L., A, L., Silva, L., Bell, L., Zhang. L.,, Guo, L., Yu, L.,, Moshkovich, L.. Wehrstedt, L., Khabsa, M., Avalani, M.. Bhatt, M., Tsimpoukelli, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M., Ke- neally, M., Seltzer, M. L., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M. Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M. Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Laptev, N. P., Dong, N., Zhang, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar, O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj. P., Liang, Q., Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li, R., Hogan, R., Battey, R., Wang, R., Maheswari, R., Howes, R., Rinott, R., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma. S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyighe, S., Chintala, S., Max, S., Chen, S., Kehoe, S. Satterfield, S., Govindaprasad, S., Gupta, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Kohler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayi, V., Montanez, V., Mohan, V., Kumar, V. Mangla, V., Albiero, V., lonescu, V., Poe- naru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W., Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wang, X., Wu, X., Wang, X., Xia, X., Wu, X., Gao, X., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y., Yu, Wang, Hao, Y., Qian, Y., He, Y., Rait, Z., De Z., Rosnbrick, Z., Wen, Z., Yang. Z., and Zhao, Z. The Ilama 3 herd of models, 2024. URL https: /ttp::/arxivvvvo .rg.v.orrg/aaaa. abs 2247.21783.
10

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Fan, Z. and Wang. Z. Spectra of the conjugate kernel and neural tangent kernel for linear-width neural networks. Advances in neural information processing systems, 33: 7710-7721, 2020.
Lin, Y., Tan, L.., Hao, Y., Wong, H., Dong, H., Zhang, W., Yang. Y., and Zhang, T. Spurious feature diversifica- tion improves out-of-distribution generalization. arXiv preprint arXiv:2309. 17230, 2023.
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic for- getting in gradient- based neural networks. arXiv preprint 1rXiv:1312.62. 2013.
Lin, Y., Lin, H., Xiong, W., Diao, S., Liu, J., Zhang, J., Pan, R., Wang, H., Hu, W., Zhang, H., Dong, H., Pi, R., Zhao, H., Jiang, N., Ji, H., Yao, Y., and Zhang, T. Mitigating the alignment tax of RLHF. In The 2024 Conference on Empirical Methods in Natural Language Processing, 2024. URL https: /arxivvv. wewiiiorrgg/aoooo arggabs/2300 6256.
Hansen, I.. K. and Salamon, P. Neural network ensem- bles. IEEE transactions on pattern analysis and machine intelligence, 12(10):993-1001, 1990.
Lu, Y., Blanchet, J., and Ying, L. Sobolev acceleration and statistical optimality for learning elliptic equations via gradient descent. Advances in Neural Information Processing Systems, 35:33233-33244 2022.
Hao, Y., Lin, Y., Zou, D., and Zhang, T. On the benefits of over-parameetrrizaioo for out-of-distribution generaliza- tion. arXiv preprint arXiv:2403. 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring massive multitask language understanding, 2021. URL https: //arxiv.. org/ abs/2009 2009. 03300.
Mallinar, N., Simon, J. B., Abedsoltan, A., Pandit, P., Belkin, M., and Nakkiran, P. Benign, tempered, or catastrophic: A taxonomy of overfitting. arXiv preprint arXiv:2207.06569, 2022.
Hu, E. J., Shen, Y., Wallis, P., Alwel-llnniiie -hu, Z., Li, Y., Wang, S., Wang, L., and Chen, Lora: Low-rank adaptation of large language models, 2021. URL httph:tt/////..... //axxi v. w.org/aba abg/ 2106 09685.
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, vol- ume 24, pp. 109-165. Elsevier, 1989.
Huang. Y., Zhang, Y., Chen, J., Wang, X., and Yang, D. Continual learning for text classification with information disentanglement based regularization. arXiv preprint arXiv:2104.05489, 2021.
Opitz, D. and Maclin, R. Popular ensemble methods: An empirical study. Journal of artificial intelligence research, 11:169-198, 1999.
Koltchinskii, V. and Lounici, K. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli, pp. 110-133, 2017.
Perrone, M. P. and Cooper, I.. N. When networks disagree: Ensemble methods for hybrid neural networks. In How We Learn; How We Remember: Toward An Understand- ing Of Brain And Neural Systems: Selected Papers of Leon N Cooper, pp. 342-358. World Scientific, 1995.
Krogh, A. and Vedelsby, Neural network ensembles, cross validation, and active learning. Advances in neural information processing systems, 7, 1994.
Polikar, R. Ensemble based systems in decision making. IEEE Circuits and systems magazine, 6(3):21-45, 2006.
Kumar, A., Ma. T., Liang, P., and Raghunathan, A. Cali- brated ensembles can mitigate accuracy tradeoffs under distribution shift. In Uncertainty in Artificial Intelligence, pp. 1041-1051. PMLR, 2022a.
Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P., and Cord, M. Diverse weight averag- ing for out-of-distribution generalization. arXiv preprint 2022.
Kumar, A., Raghunathan, A., Jones, R., Ma, T., and Liang, P. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202. 10054, 2022b.
Rokach, L. Ensemble- based classifiers. Artificial intelli-
gence review, 33:1-33 2010.
Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseqa: question answering challenge target- ing commonsense knowledge, 2019. URI. https:
Li, Y., Yu, Z., Chen, G., and Lin, Q. On the eigenvalue decay rates of a class of neural-network related kernel functions defined on general domains. Journal of Machine Learning Research, 252)) 1-47, 2024.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Lin, J. and Rosasco, L. Optimal rates for multi-pass stochas- tie gradient methods. Journal of Machine Learning Re- search, 18(97):1 2017.

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussemot, L.,, Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M.. Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsit- sulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J.-B., Neyshabur, B., Bachem, O., Walton A., Severyn, A., Parrish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A.. Brock, A., Coenen, A., Laforge, A., Paterson, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C.,, Choquette-Chone
tion with applications in data science, volume 47. Cam- bridge university press, 2018.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965- 23998. PMLR, 2022a.
C. A., Sinopalnikov. D., Weinberger, D., Vijaykumar, D., Rogoziñska, D., Her- bison, D., Bandy, E., Wang. E., Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F., Rasskin, G., Wei, G. Cameron, G., Martins, G., Hashemi, H.. Klimezak- Plucińska, H., Batra, H., Dhand, H., Nardini, I., Mein, J.. Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, J., Iljazi, J., Becker, J., Fernandez, J., van Amersfoort, J., Gordon, J.. Lipschultz, J., Newlan, J., yeong Ji, J., Mohamed, K., Badola, K., Black, K., Milli- can, K., McDonell, K., Nguyen. K., Sodhia, K., Greene, K., Sjoesund, I.. L.. Usui, L.., Sifre, L., Heuermann, L.., Lago, L., McNealus, L., Soares, L. B., Kilpatrick, L.. Dixon, L., Martins, L., Reid, M., Singh, M. Iverson, M., Görner, M., Velloso, M., Wirth, M., Davidow, M., Miller, M., Rahtz, M., Watson, M., Risdal, M., Kazemi, M., Moynihan, M., Zhang, M., Kahng, M., Park, M., Rahman, M., Khatwani, M., Dao, N., Bardoliwalla, N., Devanathan, N., Dumai, N., Chauhan, N., Wahltinez. O., Botarda, P., Barnes, P., Barham, P., Michel, P., Jin, P., Georgiev, P., Culliton, P., Kuppala, P., Comanescu, R., Merhej. R., Jana, R., Rokni, R. A., Agarwal, R., Mullins, R., Saadat, S., Carthy, S. M., Perrin, S., Amold, S. M. R., Krause, S., Dai, S., Garg, S., Sheth, S., Ronstrom, S., Chan, S., Jordan, T., Yu, T., Eccles, T., Hennigan, T., Kocisky, T., Doshi, T., Jain, V., Yadav, V., Meshram, V., Dharmadhikari, V., Barkley, W.. Wei, W., Ye, W., Han, W. Kwon, W., Xu, X., Shen, Z., Gong, Z., Wei, Z., Cotruta, V., Kirk, P., Rao, A., Giang, M., Peran, L., Warkentin, T., Collins, E., Barral, J., Ghahramani, 7., Hadsell, R., Sculley. D., Banks, J., Dragan, A., Petrov, S., Vinyals, O., Dean, J., Hassabis, D., Kavukcuoglu, K., Farabet, C., Buchatskaya, E., Borgeaud, S., Fiedel, N., Joulin, A., Kenealy, K., Dadashi, R., and Andreev, A. Gemma 2: Improving open language models a practical size, 2024.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7959-7971, 2022b.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang. K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T.. Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Liu, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., Guo,
Z., and Fan, Z. Qwen2 technical report, 2024. URL https: / /ttp/v/wvvvv/.//////////////.av. wwwpiiiv.org///////eeeeo 2ot.. 10671.
Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. W. Py- hessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pp. 581-590. IEEE, 2020.
Zhang, T. Mathematical Analysis of Machine Learning Algorithms. Cambridge University Press, 2023. doi: 10. 10.1017797810099993077
Zhang, Y., Chen, C., Ding, T., Li, 7., Sun, R., and Luo, Z. Why transformers need adam: A hessian perspective. Advances in Neural Information Processing Systems, 37: 2024.
Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, L., Shojanazeri, H., Ott, M., Shleifer, S., et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.1 11277,, 2023.
URL https: http///axvvvvvv. org/ abs 240//2208..o 00118.
Tian, J., He, Z., Dai, X., Ma, C.-Y., Liu, Y.-C., and Kira, Z. Trainable projected gradient method for robust fine- tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7836- 7845, 2023.
Zheng, I.,, Chiang, W.-L.,, Sheng. Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging alm-as-a-judge with mt-bench and chatbot arena, 2023. URL https: //arxiv. //arxiv.rg//abs: htt.bb///366...50
Vershynin, R. High-dimensional probihaaallinnaaoohllllllll probability: An introduc-
12

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Zhou, Z.-H., Wu, J., and Tang, W. Ensembling neural networks: many could be better than all. Artificial intelli- gence, 137(1-2):239-263, 2002.
13

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
A. Experimental Details
A.1. Hyperparameter Searching
We have conducted hyper-parameter searching on the fine-tuning process and the ensemble process. We fine-tune the models with global batch size of 16, and an epoch using Adam optimizer on 8 GPUs. To select a suitable learning rate and penalty, we search the learning rate on (5 10 9 2 x 10-6 10- -), and penalty coefficient À on {10-2.55 2.5 10 x 10-3 10 We also search the ensemble weight uniformly on {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}.
To preliminarily validate the performance and choose the hypeer-parameter, we have carefully curated instruction-following dataset. The validation dataset consists of multi-turn conversations between the user and the assistant, covering writing, reasoning, coding, math, STEM, and humanities topics. We prompt GPT4 using the prompt "Help me generate 3 sets of 2-turn instructions to evaluate the {category) ability of LLMs. The instructions for the second turn need to be highly relevant to the first uum. The following an example. n'nn'n EXAMPLE: (example) y vn TURNI: [tunn/ yi TURN2: [turn2 nn". where {category) corresponds to one of the categories in MT-Bench and {example) is one example from MT-Bench. In this way, we obtain validation dataset that is highly similar MT-Bench. Specifically, our validation dataset contains 600 samples, evenly distributed across the categories in MT-Bench. We then represent the performance using the loss calculated on the validation dataset.
A.2. Implementation
We implemented our fine-tuning code based on Huggingface Transformers and Accelerate libraries, where Fully Sharded Data Parallel (Zhao et al., 2023) is utilized for model parallel training and acceleration. Our training and evaluation are conducted on 8 NVIDIA H100 GPUs.
A.3. Extension to LoRA
Methods
MT--Bench
Commonsense-OU QA
MMLU
DiffNorm-Penalty + Ensemble
5.85
74.49
63.97
LoRA
5.83
73.71
65.29
LoRA EEsemble
5.99
73.87
65.31
Table 2. Performance comparison of different LoRae- methods.
We also conduct experiments with LoRA (Hu et al., 2021). Specifcally, set 32, 32, dropout 0.01, and target modules to q-projection, v-projection. The results are shown in Table 2. We first observe that LoRA can mitigate overadaptation as well but tends to forget more in certain benchmarks, such as Commonsense-QU in comparison with DiffNorm-Penalty. On top of that, it is observed that further ensembling with the pre-trained model yields additional performance improvement all benchmarks. Such results highlight the benefits of ensemble methods.
A.4. Variance of MT-Bench
We also examine the variance of the MT-Bench by fine-tuning the model trials with different seeds, under Norm-Penalty and ensembling with r = 0.8. The results are shown in Table Overall, we observe a standard variance of 0.06, which is sufficiently small in comparison to the score gaps in Table
Trial
Trial 2
Trial 3
Trial 4
Trial 5
Standard Deviation
5.84
5.91
5.95
5.96
6.01
0.06
Table 3. Variance estimation of ensembled Norm- Penalty with 7 = 0.8 on LLaMA-3-88
14

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
B. Empirical Validation for Theorem 5.1
To validate Theorem 5.1, we first utilize artificial datasets, where we construct pre-trained and fine-tuned datasets based on 4 diverse groups of parameters, respectively. Specifically, consider 4 cases for different eigen- value parameter ? and the size of pre-trained set n: (a) ? -1.0 n 40; (b) 7 = = n 40; (c) 7 For each case, we set data dimension as p = 10000, the size of test data 1000. We generate a pre-train dataset and a fine-tune dateset from two normal distributions (0, Σ1) and (00 22), respectively, where Eσ has eigenvalues A1 = 1, 12 =... = Xp = n- n- and Σg has cigenvalues A = 1, A2 _.. = An =n An-1 = =... = Ap = 0. The ground-truth parameter for the pre-train and fine-tune Og + an and 0g + a2, where 110o112 = I, a, No, 4,), and a2 (0, 0.121p). The variance of data noise is 0.1². After obtaining the pre-trained estimator 01, we fine-tune the estimator on the other dataset to compute the "min-norm" estimator ég according to Equation (4), and the estimator with regularizer 02. We tune the hyper-parameter within a range and choose the À that achieves the best excess risk, which is À 0.0001. Finally, we calculate group of ensemble estimators 0% with t ranging from 0 to 1, and plot the curve of the error on the pretrain task versus the error in the fine-tuning task for the group of 0% in four cases in comparison with the fine-tuned estimator with difference A, the "min-norm" estimator and the 0х in Figure 4. According to Figure 4, the performance curve for the ensemble estimator 0% achieves better trade-off on the two tasks compared to fine-tuning estimators, which aligns with Theorem 5.1.
n 40; (d) 7 = 1 n 60.
Additionally, validate the performance only on the fine-tuned task, we also consider the four settings mentioned above. To simulate the realistic situation that it is difficult find the best X, and
only tune it into a small range, take A = le 7. Finally, we calculate a group of ensemble estimators 0g with ranging from 0 to 1, and plot the curve of excess risk for the group of 03, in four cases comparison with the pre-trained estimator, the "min-norm" estimator and the 8 Figure 5. The figure implies that if we tune the ensemble parameter 7 to the optimal, the ensemble estimator 0% performs best, and then the performance decreases in the order of the estimator with ridge regularization 02 the "min-norm" estimator 0g and the pre-trained estimator 01, which aligns with Theorem 5.1.
15

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Performance of Ensemble
Parformance of Ensembla
Gnsemate 0-00001
Ensemble 0.0001
Artbrin
Protrinn.
Pretrais 1ed6
Pretmun leco
Probian 10:03
matrain I6-C5
Prebrina 0.0001
Prdtrain - 0.0001
Protrai Tack
1-
Kizc Sumc Sed
fone Task
(a) 0.025, 40
(b) 0.004, 71 40
Performance
Performance Ensemble
Broemote 3,0001 0.0000
Ensemble A 0.0003
Pestramn A m0
Pretrain =
Pribran 1047
matrain Iec7
Prebrain. 1e-66
Pretrain leco
Pritrairs 0.0001
Pretram Seak
Tack
(c) 7 0.017, 71 60
(d) 7 = 0.0022, 60
Figure 4. Performance of Ensemble with dimension p 10*
16

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Performance of Ensemble
Parformance of Ensembla
Ersemble Risk:
Protrain Bisk
firetune (aamm0)
enieeme
0.4
0.6
0.6
3.0
Taa
(a) 0.025, 40
(b) 0.004, 71 40
Ensembble
Performance Ensemble
Ernamble Pask
Ensemb
Fretram Risk
Fretrain Rick
clure Mirk m0)
Nicctu n2A dam-th
Pir coun Rutt 10m0020000
Fristune hak Bam- (70
1.0
30
0.0
02
0.8
1.0
0.6
0.6
3.0
(c) 7 0.017, n 60
(d) 7 = 0.0022, 60
Figure 5. Performance of Ensemble with dimension p 10*
17

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
C. Proofs
C.I. Notation and Constant List
Before the main proof process, we denote several corresponding constants and notations in Table 4:
Symbol
Value Expression
max (1 16 In of 551e332 51c}
c
256 (162e)'
C1
max (L/c' -
C2
8(162e))
C3
A
the le-th eigenvalue of matrix 21 ie, (2)
A
the l-thh eigenvalue of matrix E, i.. (2)
Ay
Tk
A4+1
Table Constant and Notation List
The definition of Fu is the same the definition in Bartlett et al. (2019). In Bartlett et al. (2019), the critical index s (6) for given b > O is defined as
(b) == inf k 2 0: : nk 2 bn}.
(12)
In our data settings, without lose of generality, we choose b = 1, and obtain the critical index s1(1) = -k" in E as well as Σ.
C.2. Excess Risk Decomposition
The detailed analysis is start with composition for excess risks. First, the estimators mentioned in main text could be expressed as:
0, - -xa +c),
82 81 + ("X""
= =11-7"0))1101100003310----0000,0 +2)) x
= X|00
0, + - --
= XXXX" -XX"' +
XXX 1(0 +z) XXXXX" +nATI
= -xx" - - XXXI)
[[I-X"(X"" -XX""""XXX"""""""""XX""""""""""" + X"XXX" +nd
+ +[T " ("X*" + 2
σ = (1 - -
[X" - -x
X + -
Rx (XX* +nX XXX X]0.
Xoa + -X +n -XX2
" (XX" +
18

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Then focusing on the excess risks on Task 1 and Task 2, i.e.
pree (0) ,,,,,,,,,,,, aa (0 0)² 0), (0) ag (0= 0),
we have the following results:
+
Lpee (02) 102)=002000000 1-
+ + x
(02) = 00 ||
+ +
+<=r[[""((**++ +
-xx(XXx"=x11-XXXX"" + (E
+02tr[(XX*+*))) "XXX"
(0) 0.14
+ "I-X"("""""x+=""""X""" + XXxxxxxxXXXXxxxx-xxXx
(
" (XX"m -nd
+ + -
and
+x'"(("""))xxx"""
- ",
+ "x""(1=""XX"*xxx) (
"
+ -M)
- +nAI) XXII -X (XXXx)x-XXX x10.
+ ((
+c20r(11 -xx""
=x - -x -nAA)-'((''
+x -
19

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
C.3. Term Bounds Estimation
After obtaining the expressions about different terms within the excess risk, we could estimate the related upper and lower bounds now.
C.3.1. TERMS CORRESPONDING TO 0g
All of the excess risks about 01, 82, 0x and 03 contain a term related to d.. Here we could obtain their upper bounds for Task 1 as
1 XXXX"" 3 x 11 X"XXX"XX"" 10.1311 _ X|1
119 - X
-
"XX1--"((X"""XXXXX"">0>>>>>> < |0. Σ x1
" - -
+ + -XI==
- XXXX x/0 S 10.1/1- 1 X11
72
= ( X - - XXX
-xxx)
0 - S
-
and for Task 2 as:
6. - -" -10
XXXXx)-- [ - -- 10.1111 -11x|12
< "( -0,00 -
|11--""|AX"--3000==""""""""""""" XXX" + XXxx
s - - XXXXXX"""'"00X0--"""")> x]0 - 10.123 X
X (XX* - - -xxx))
"0 - 10. 12 x
which implies that we just need to upper bound the following two terms
1 _
|
_ x 1112 S 15 E112 + 11
x" X|12.
7
72
n
From Condition 1 and 2, we have
- -リー S maxt% 7
and induced by Lemma D.7, with probability at least 1 - e -n we have
| 1 XT X|| S n 1-3
(13)
Combining the two results above, we can also obtain
| 1 xT S maxfy, 7} + n -5
(14)
20

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
C.3.2. TERMS CORRESPONDING TO {1
For these terms corresponding Çi (02), (0x) and Epre could approximate their upper bounds as:
hp Gitr[[- 11-V1 S (Ir(E))
G - -XX"XX")"11=+""0XX"""""""""""""" -
- (ttr(}),
Gtrl[T-- "F +nA
1 X-X"(XX""-Xx++XX" +
((r{E).
Similarly, for the terms in (0)) Ln (0x) and L (0.) we can obtain their upper bounds as:
- - -
<(tr((xx")-xXXx"',,
(tt(((x"))'xX XXXX" -X -X11
< xxx((X
x - -
s <Gtr((x""=-xxxx")
which implies that for estimating the upper bounds of these terms, we just need to upper bound the following two terms:
Gir(E), x(lx((xx*)-xxx"'
The first term could be expressed as and we just need to approximate the second term. First, recalling the decomposition Σ E, Aree, we have
(rr(2]= = Ci (k' +m))
(15)
XXT Σ Aaat' XEX = Σ Aaaef
(16)
in which
Xe: independent -u-Suagaussian random vectors in R" with mean 0 and covariance I. Then we will take the following notations in further analysis:
(17)
A=XX², A = Mizie A = Σ AzZzeT.
(18)
i>k
ig
Using Woodbury identity, we have
-(1((Xx"--XXX") = Σ XI 12
Aodizi A
(19)
Σ XxxIT(XXT)
=
X12 1_
For any i == 1,..., k², we have
2
11 23 11 pk+ (Ami)
T AA- <
T A (112, A
(20)
(n(A))))
21

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
where La is denoted the subspace in R" related to the k* eigenvalues of A_;- Considering Lemma D.1 and
Lemma D.2, with probability at least 5e-n/c, we have
41 S Am S Mk +1 S C1AR-17%
| S c
| 2/e33.
where 61 C22 C3 are constants only depending on 0z. The results above imply that
C1e2n
2 2/
so with probability at least 1 - n// we have
A 1+ X, A
Goondi/(Ax 1+ A11122/ he102n/22k++17>> (Ax-+17k-)
k²
k.
(21)
where the last equality from Condition For the remaining part, considering Lemma D.I, with probability at least 1 2e nncc we have
Eien- XAMlE (XX²)
S Eisk AcAel Eist XAME
Σ Xx S
S S
11-+
and further considering Lemma D.6, with probability at least - 22 2n/c, we have
n Σ X2A3/00
Σ xMMl& sn Σ 1
T + 20 max
ny Com
-nnyy + 20,, max
2n
where the second equality is from Condition 1 and the last inequality is from Condition 2. This result implies that with probability at least 1 -
n/c, nn/e, we have
G2mpYY
e22npry 2c/npy P
Σ Au (xx") $2*
(22)
17
Combing the results in Eq. Equation (19), Equation (21) and Equation (22), with probability at least 1 - 1n/en/ we could obtain that
202nān
x sc k" k
(23)
p
C.3.3. TERMS CORRESPONDING TO
Similar to the analysis above, these terms corresponding to ar in excess risks on Task and Task 2 can be upper bounded by
(xr((xx")=xxxx")
respectively, and the estimation of its upper bound is similar to the previous item. Here we first consider the second item, by Woodbury identity, we have
- Σ T.A (XX=F(XXX))
(XX*) 22
(24)
No A2,24
+ E
Aedie (XX")
+ A
i>k
22

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
From the results in Equation (20), with probability least 1 Dn/c, for any ,,... k", have
Geen
of A S
] 2
n
(de++1rk=)2*
41332+417K
which implies that
A2,2
k'
So
1 +
+
k'
< 0²²
(25)
2 2
S A
o² c
1 X o²
Σ A
k²
k* n
n²>?
n
where the second inequality from (a + b)2 a² + 12 while a, > O, the third inequality is from a² + 12 2 a², and the last equality is from Condition 1. And for another part, considering Lemma D.I, with probability least 1 -e/n/e we have
of (xx²) Eisk XAde Pn (xXT)2
cj Eish X
and further considering Lemma D.6, with probability at least
26 2/-n/o we have
nd
Σ SAlLE Σ Xede 20 max 1
Σ X333/0
1>*""
npry + 20 max
nry
72
np
2npyy
where the second equality is from Condition and the last inequality is from Condition 2. which implies that with probability least 1 nn/e we have
n/c,
Σ Xxxx I 22 S>
npry x 2-
= =0 rip p2
(26)
where the last equality is from Condition 1. Combining the results in Equation (24) Equation (25) and Equation (26), with probability least 1 1/2en have
(xx((x'))=xxxx s0² (ctoes
k" + 20 np
(27)
12
The analysis for the first item is similar, which implies that with probability at least 1 - 10c -n/2e, we could obtain that
(xx((xx))xXXx') s0² (At2e + +207 n2
(28)
C.3.4. TERMS CORRESPONDING TO &2
In Cpee to approximate the upper and lower bounds of terms related to S2, we need to estimate:
(XXT - = (2tr(
+ 2X
23

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
for any A 0. And in La. for the terms related (2, we have the following results:
C2tr Gatr((1-X (2tr((T- - - -
= - - + (EXT)
Sar(E(1X''(X33 + 2 (an(((1-X'(xxx)))
- -G(r(E- - -xx0))
( + 5 <(ttr($(I- - +XM' 18)
=Gtr/EE - (2 + nAT) "XX"),
G + n))) > -xx=)(1-x""(xx)-133
so here we need to estimate the upper and lower bounds for term
( +
for any À 2 0. The analysis is similar to the analysis above, recalling the decomposition E= = E, AxeeeT we have
XXT = Σ Eaa
XEXT = Σ Ratt.
in which
5 Xei,
are independent og-subbauussian random vectors in R" with mean zero and covariance 1. So we take the following notations in further analysis:
A= Ag Σda À== Σ A
(29)
i>kk
ift
Starting with the analysis of (2tr((XX) -n }. we consider its upper bound firstly. Using Woodbury identity, we have
(2 +n\l) "
= 62 Σ X (XXT +nA1) -zz
(30)
k"
X (Ā nA 15
(XXT +
+
1+ 1x (A + ndl) -1,
12 1
For any i == I,..., k., we have
|
(Ā_ =2)) 12 -255
Mx (A_,)_) nx
(31)
TI 37 11 A4 +1 (Ams)
/ 1 > (1I و ") (A
[II 5
where L, is denoted as the subspace in Ra, related to the n - ks eigenvalues of Ā_, Considering Lemma D.1 and
Lemma D.2, with probability at least 5e n/c we have
6 Ax.477 S (.,)) s me pe (1 A E < c
n/C3:
(32)
24

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
where 6 02,02 are constants only depending on ow-
The results above imply that
o (A-i+ + nAT) 2 1 + +nd
C1 1
(A + n
n
+k + +nd)
so with probability at least 1 - 5e n//o we have
X (A_ + - E (A -
x a + nd)
S Ex,
i= 1 + / + nd)
=k* a02nn(in + +n)
(33)
1+ (p> + nX)
< k* min
C1c2n
C102
4 +M C1
where the last inequality is from a + 2 maxfa, b). For the remaining part, considering Lemma D.1, with probability at least 1 - -e n/c, we have
Esst X31 Eise X1E Pr (xXT) + n2
X (XX* =
21 + nd
and further considering Lemma D.6, with probability at least
26 n/oo we have
n n
X S Σ X + 20 max
X//c
1>
i>
m2
np
niñ2 + 20 max
72
S
2njoy
where the second equality is from Condition 1 and the last inequality is from Condition 2. This result implies that with probability at least 1 - 4e/n/e n/c, we have
2m2mpy?
20fnj7² P7 +n S 26 min
/y² { no, X f
Σ XXXX +nAl) < E>K
(34)
41 +
Combing the results in Eq. Equation (30), Equation (33) and Equation (34), with prohability least 1n/2n/ we could obtain that
(otr( + - C2 kk" minf
Cio2n + iy c102 + + + 20 min noy,
C1o
C1
p7 1)
(35)
A
Then we turn its lower bound. Considering Equation (31) for any index i 1,..., 0c,
H4 4 & pk +1 (XX)) & f
always satisfied, with probability at least 5e -n/c we can get lower bound as
(A -
X2n + nd)
CiCs + (A + nd) 1 X (A_; + -1z)
(36)
X/n/(u- + nn)
1
Am/ nd) > 0,
25

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
and for the whole trace term ( we have
- due Lemma D.5, with probability at least nn/e
nxT) n - 2 (A- + n
Gtrl(XX +
22
C2
Σ
/tne/ (At0 +T= + +nd)
201
XiT1/ (A The n2)
(37)
C
Σ min
A
Ai: nd?
C3
A
2x-417k-
C2
min 2 my 1)
k*
min
x 1 + (p - k*)
6c1c3
where the first equality is from Woodhury identity, the first inequality is from Equation (36), the second inequality is from the fact
(a + bbb)+ max 6, b,c))= = 3
a, b,b,c> 0,
and the last equality is induced from Condition and Condition 2. And combing Equation (35) and Equation (37), we could consider different situations with respect to the value of X.
While x S /r/n, we have
cie2k"n
(atr((XX" +n\l) -XX-" Se
+2cînz c102n + +
(38)
ny k*)
c2tr(((X] +
k²
63
And while À 2 in/n, with high probability, we have
cie2k
209
( + -AI)
Se
C102
+
A
(39)
62
T. 13 + 70 2- k*)
(2tr(([XX +
min
6cg33
A
Then we the term (2Lr (X -XX X
While we have nd S /n/e,, with probability least
1 5e n/c we have
nd < pnn (XX*)
- n\I = XX
XX + nAt 2X
which implies that
6 -
(xr + 2X
(40)
as we also have
+X = - + XXX
< (
(41)
Combing both of the two results above, just need to estimate the term COt - Using Woodbury identity, we have
+ - Co EAA + nAd) -2,
and recalling the identity
( + - = 6 Σ X2 [XX* +n
comparing such two terms and Condition 1, we have
N = X3 Vi= 1.....P p.
26

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
which implies that
(2 + - (atr[(XX* + +
(42)
Then considering Equation (38), Equation (40) and Equation (41), while nd S p7/c1, we have
c102k"n n
2 + -X S2
+ +222n7
C102 + py
(43)
C2
+ ny k*)
2 + XXXXXXX
240
p
C.3.5. TERMS CORRESPONDING TO
Finally, for the terms related to o, we need to estimate the bounds for the terms (Xxr((XXX)))
and (xxr((XX*++++ +
nAI) -X ΣX} in Cree. and the terms (XX"((XX ΣX} and @etri (XXX + - in Ct. i., for the
terms (XX + - XXX" and @irr((XX* X + n\I) - "XXXX in which À 2 0.
Firstly, we consider the approximation on identity follows:
+ - For its upper bound, we could take Woodbury ,x + nn) "
(XX* + +nI1)
(44)
A + n1l) m))
+
(XX*
+ nXl) 22
3 (A_,+ -
i>
According to Equation (31) and Equation (32), with probability at least
5c n for any index i = 1,..... k", we have
( (A S - 1222
{AA + +
n2
(45)
( + nn)e
( n2d
which implies that
X (A=+ md1) 2/
R"
/cjoon/(xx + +n)²
+
11
+ Xm/ (11c) + c103nA)]2
NotereIn
<0²2
22 + Gi
22 + ciczn2X2
1=1
(46)
X X²
S 6 2
min
2 ++ k-
1
82 c min
X2
where the third inequality is from a² 12 +c2 > max 12, (2), and the last equality is from Condition and Condition 2.
As for the remaining part, considering Lemma D.I, with probability at least - 2e n/c, we have
Σ X2 + nAA) -2 S ΣiEr XI
Mr, XX n S o Σise- 111 2
Eon
12 22 + 2
and further considering Lemma D.6, with probability at least 1 - 2e n//c, we have
nua
Σ A
X n Σ X + 20, max
Xt/0
n
c
$>
i>
ny²
mp
n 2 + 20 max
2
S
2npp?²
c
27

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
where the second inequality is from Condition 1 and the last inequality is from Condition 2. It implies that with probability at least 1 4e-n/e we have
Σ XXX + nAT 2 S 2 p222 + n₂ソ² I 222o0²
npy
min n 572
(47)
p nd3
Combing the results in Equation (44), Equation (46) and Equation (47), with probability at least 10e we have
,- +nA x000 (oo2cik** min
1 1 n
X2 + 2c
min n ~n² p nx
}
(48)
Then for the lower bound, each index i = 1, 00, we have
a (A nAl) 22 >
(A:
which implies that with probability at least 1 - -5-5-////
+ +nd) [] 1+ 1 + nA1) -
N (Ā + n
2 12
1 2 (A+ + 111)-112)
(49)
2
1
ndo
>
O2n
(1c +1k+ ) + nu
where the last inequality is from Equation (31) and Equation (45). Then for the whole term, due to Lemma D.5, with probability least 1 n//c, we have
[ (A + + [1
pur((XX++ XXXX) J2
23 27
(A ndl) 22,12
nd.
202
Cica(Ax- +kk + nd) + nd
0²
1, X2
n²
(50)
min
18272205n
X2
1 n²
(p k*) min(1, (1, 72 n2
k- min
(1,
x2 p2 )
1822
X2 p2g2
1 min (1,
x2 } + ( k* min
1
{ x2 p7
n²
1)
18c20203n
where the first inequality is from Equation (49), the second inequality is from the fact that
1 9
(a + b bb+ -)) -22 max b, c)) - min (a minna--2 -., 2), va,b,c>0 0,
and the last two equality is from Condition I and Condition 2. Combing both Equation (48) and Equation (50), we could also consider the following two cases.
While x < in/n, with a high probability, we have
(fozoik**
,- -XXX"
S
+
22
P
(51)
1
k*
alp
kz)
2
+
18
28

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
And while A 2 in/n,, have
σ²tr((XXX +nAI) E
(do2cctt* minf
1
} + 20
202 vigo
n' X2 nd2
(52)
k' min(1,
1
3 Pp--K - k*)
-XXX /² 12
11
nd?
Then we turn to the term x + Similarly, using Woodbury identity, we have
jx - = 32 Σ A (XX" + +ndI) 2z0.
as well as which implies that The result above implies that while À & 17/n, with high probability, we have
8² pr +NM)) = 82
(XXT T +
comparing such two terms and Condition 1, we have
X X3 Vi= I.....p
σ + nA1) "XEX'= = =r -
(53)
($tOre3**
ou + nA)) - <0²
+ +20 12
n
p
(54)
1
k*
n(p k*) 1
+ 8
+
186G
n1
And while X 2 /n, we have
82g2
σ2 + 50
(Cs2zzh** minf
+ 20²
X2
n22
(55)
k
1
72(p - k*)
-XXX" 2 /
-
-min
1,
18
x
nd
C.4. Analysis on Lt.
From Condition 2, we have the following results:
o max(7,7] n- (1-8)/2) < ((22M)),
( < O(Gm))
0 (n (pp\(p²>))) (C2(1 + my)) x O((2T)). O ( + n/p) x O({27))
< (((2p)),
Then comparing Equation (14), Equation (23), Equation (27), Equation (38), Equation (39), Equation (51) and Equation (52), we could obtain that the excess risks on different estimators always dominated by terms related to (2 and /². So for the ridge regression, we have
(r,) xe - + (
- = f(A),
then take derivative with respect to X, could further obtain that
f'()) 2n(G2nd - -
29

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
which implies that the optimal choice of À nearly equals to
X σ²
Onn
As 0₂ = 0g i 0 and the optinal value X' > 0, we have
Ca (x)) S Cm(2)) VO A 2X.
(56)
And according to Lemma D.1, considering Condition 2, with probability at least - 2e 2// we have
1 ( (XX 2 fー
6²
c1
C2'
which implies that
x² x( - Galx(((x")-XXXX") - -
<
so we could obtain that
(b)) La (01).
(57)
Combing the results in Equation (56) and Equation (57), we could finish the proof of the first item in Theorem 5.1. Finally, while choosing 0 S X, according Equation (38), Equation (39), Equation (51) and Equation (52), we have 82 tr + nA) -X x (2tr(((XX +nd - x (tt
And for the excess risk of ensemble estimator, we have
(n)) = (2tr - 7X (XX* +
then take derivative with respect to т, we could obtain that
gT( = 2T(2tr ( nN
+ -
2 n\/) -
which implies the optimal choice of F is as
(2 + +
7(1) (Xr((XXT + +
-XX
Ftx - "XEX")
S /2 (n(2), we can obtain that 0 5 (A) 1, due the fact that
6TT(("X" + and "X + - + + 2X
-2arr((X + +n) -XXX"
==f((X"" + -nd) 2 (51+ (xxx" + (XX - +orTT+nNNi + {EX*
= =ir( + n\1) - (521+ - x( +nd)) {EXT]
= - (anA))r(((XX"" ((Xx"++\(X -XX0) 2 0.
+nAI)
So we could draw the conclusion that for any T((A) ST < 1, we have
(a())) La (,))),
(58)
which finishes the proof of the third item in Theorem 5.1.
30

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
C.5. Analysis on Peee Lt,
Similar the analysis on La. based on Condition 2, could compare Equation (14), Equation (13), Equation (23),
Equation (15), Equation (27), Equation (28), Equation (38), Equation (39), Equation (43), Equation (51), Equation (52),
Equation (54) and Equation (55), and obtain that
Cpre (0x))) 1atr[(1- - - + + XXXI)
+ (XX" x + + "XEX"
Salr(1-X'(XX) + xx -
+X +
= -(A),
where the second equality is from Equation (42) and Equation (53). Taking derivative with respect to A, we have
(A) = 2ntr 2 + nAT) ((Gm) 2 XxXX1 XEX"
which implies that while x < 2X' = 27 2 ((1(2), we could always obtain
(A))) 0.
So for any < S 2X, we have
(0)) + Ca < Cpre (02) + (02).
(59)
And with this range of we could obtain the similar result as:
Cpre + Cn (0T)
1 +r
-X
J(],,,
and taking derivative with respect to r, we have
J'( = 4T(2tr +
-XX
2(2tr( (XX -
which implies that the optimal choice of 7 (2)) And while A S 2X' we have
4TG(T((XXT + nd) -XX 47 + +nA) - (X -XXX")
- nAI) - + - (GnA) > 0,
which implies that 0 0 r'()/22 < 1. So we could further obtain that for any 7' /(())/2 2 < T < 1, we have
Spre (0)) (0g) (0)) C0))
(60)
Combing the results in Equation (59) and Equation (60), we could finish the proof of the second item in Theorem 5.
D. Auxiliary Lemmas
Lemma D.1 (Lemma 10 in Bartlett et al., 2019), There are constants b, c 2 such that, for any k 2 0, with probability at least 1 - ē,
1. for all i 2 1,
(k+1(A-)) S (x+1(A) S (Ax) <a A + A1tin);;
32k
31

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
2. for all Sisk,
(n(A) 2 2n((A)))
(nx)
Σ> N3 - ci A
C1
3. rf 2 bn, then
1 Ak+1T& (n() 1M1 S AAAR+ITK C1
where 01 > 1 is a constant only depending on 0z-
Lemma D.2 (Corollary 24 in Bartlett et al., 2019). For any centered random vector 2 E R with independent o sub- Gaussian coordinates with unit variances, any dimensional random subspace L of R that is independent of z, and any t > O, with probability at least 3
> 2 vnl), y 11 n 2(1626) + + Vnl),
where Ug is the orthogonal projection on L.
Lemma D.3. There are constants b, 2 such that, for any ls 2 0, with probability at least 2e :.
1. for all i 2 1,
(A+I(A + AT) M++1
S N) [ >3 + da+1m) + x;
J3k
2. for all 1 isisk k.
Mn AA(A+MI (A 2
23 Σ>
01
j>k
3. rf 2 bn, then
A + X S (Ax + AI) S (Ax + AI) S CIA+ITK
Proof. With Lemma D.I, the first two claims follow immediately. For the third claim: if 2 bn, we have that bm 5 Ejsr Az.50
(1AA AI) S C1 A1 + + c1
1 pn (Ax + X1) ARtITh +Y> Ae+ITa +
for the same constant e1 in Lemma D.I.
Lemma D.4 (Proposition 2.7.1 in Vershynin, 2018). For any random variable & that is centered, sussuaaasssann and unit variance, g2 1 centered 162ea 2 -ubexponential random variable, that is,
Exxp(A(E" - 1)) & cxp((162cx03)))
for all such X that [X] < 1/(162e03).
Lemma D.5 (Lemma 15 in Bartlett et al., 2019). Suppose that }n is sequence non non-negative random variables. and that (t:} is sequence of non-negative real numbers (at least one of which is strictly positive) such that, for some å E (0, and any i 2 1, Pr(m: > 4) 2 1 --8. 8. Then,
Pr
Σ
m
t
2 11-28 28.
2
32

---

Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Lemma D.6 (Lemma 2.7.6 in Vershynin, 2018). For any non-increasing sequence {A,);=1 of n'oonnegative numbers such that Ej>r oc, and any independent, centered, subexponential random variables {E) and any with probability at least 2
Σ A.E S 20 max
xd1, x
X3
Lemma D.7 (Theorem 9 in Koltchinskii & Lounici (2017)). Let 21,.., 2n be i..d mub-gaussian random variables with zero mean, then with probability at least 1 - 2e,
1
trace (Rez") )
trace(EzzT]
trace (EezT)
1
E22 Tf
2 1l S |E22 maxfy
π2
11
72
33


Reviews:
[{"review_id": "JS2gsb5s0f", "reviewer": "Reviewer_2w5t", "summary": "This paper studies overadaptation in supervised fine-tuning (SFT). It builds on prior work that observes that ensembling a pretrained model with its fine-tuned variant improves performance. Within an over-parameterized linear regression setup, the paper theoretically shows that this effect is due to the ensemble's ability to manage bias and variance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive suggestions! Additional experiments and explanations have been provided as follows.\n\n**The connection between theoretical and empirical results.** Our experiment on LLM and theory match in showing the benefits of ensemble in fine-tuning. As fine-tuned parameters are close to pretrained points, we can adopt a linear setting with NTK explanation. Specifically, we can approximate the neural network $f(x, \\alpha)$ as $f(x, \\alpha_0) + \\nabla_\\alpha f(x, \\alpha_0)^T (\\alpha - \\alpha_0)$. Comparing this with our linear setting $y = x^T \\theta$, the features $\\nabla_\\alpha f(x, \\alpha_0)$ can be interpreted as the input x in the linear model, and the trainable parameter $\\alpha - \\alpha_0$ as $\\theta$. Since $f(x, \\alpha_0)$ remains unchanged during training, its effect can be disregarded in this simplification. Such simplifications can provide intuitive insights and have been widely used in prior works [1,2].\n\nThe high-dimensional assumption on x characterizes the “features” $\\nabla_\\alpha f(x, \\alpha_0)$ in large models, i.e, there are several large eigenvalues and many small eigenvalues. It can be validated by the eigenvalue distribution of Hessian matrix in [3] [4]. Though bridging the gap between empirical results and theory is still challenging, this work takes a first step toward a theoretical framework, and we aim to develop more comprehensive explanations in future works.\n\n**Figure 1.** Thanks for the suggestions! We will adjust its size accordingly. Figure 1 illustrates the effects of overadaptation in fine-tuning, showing that model performance degrades with additional training epochs. This phenomenon highlights the importance of mitigating overadaptation to enhance model performance.\n\n**Figure 2, 3.** We have improved the figures accordingly and highlighted the checkpoint source of all ensembles (https://anonymous.4open.science/r/ICML2025-rebuttal-forgetting-1F68/README.md ). Additionally, interpolation weights of 0.0 and 1.0 are also included to cover the two extremes of the pure pre-trained and fine-tuned model respectively. \n\n**Extension to other ensembles.** Weight-space and output-space ensembles are indistinguishable in linear models. Our theoretical results can be directly extended to ensembles of multiple fine-tuned models with properly chosen weights, under the assumption of same-distributed K fine-tuning datasets. The model obtained via model soup has the potential to reduce variance in fine-tuning risk, as well as both bias and variance in pretraining risk. Extending to those directions is interesting and can be future works.\n\n**Discussion on overadaptation and catastrophic forgetting.** Overadaptation means overfitting on training samples during fine-tuning, leading to degraded performance on fine-tuning tasks. Forgetting occurs when fine-tuning on specific tasks reduces performance on pretraining tasks, effectively forgetting prior knowledge. As they are closely related, overadaptation can contribute to both fine-tuning performance drop and forgetting.\n\n**Sparsity assumption.** The sparse structure observed in fine-tuning tasks reflects the nature of knowledge specialization across different inputs. While pretraining involves diverse inputs encompassing broad knowledge,fine-tuning is performed on specific tasks with a narrower scope,leading to a sparse structure. For the measurement, the eigenvalue distribution of the Hessian matrix can be analyzed in LLMs using PyHessian[3] and variants of Lanczos algorithms[4].\n\n**Linear and nonlinear models.** One concern is that the simplification from nonlinear model to linear model is reasonable only in NTK regime. Our results may not be directly applicable out of NTK. In practical models, the choice of ensemble method,such as output-space average or parameter-space average (indistinguishable in linear models),impacts model performance. Another important factor is the use of input-dependent weights (router) in ensemble. Different strategies for designing routers can impact performance. \n\n**Extension to LoRA.** Further experimental results on LoRA (https://anonymous.4open.science/r/ICML2025-rebuttal-forgetting-1F68/README.md ) show that:\n* 1) Indeed, LoRA can mitigate overadaptation as well but tends to forget more in certain benchmarks, such as Commonsense-QA in comparison with DiffNorm-Penalty.\n* 2) As predicted by the reviewer, the combination of LoRA and ensemble yields further performance improvement in all benchmarks.\n\n**Implication for RLHF/RLVR.** Yes, these findings can be extended to RLHF, as observed in [5]. And we believe similar phenomena also exists in RLVR and will include relevant discussions.\n\n[1] Benign, tempered, or catastrophic: A taxonomy of overfitting.\n\n[2] Fine-tuning can distort pretrained features and underperform out-of-distribution.\n\n[3] Pyhessian: Neural networks through the lens of the hessian.\n\n[4] Why transformers need adam: A hessian perspective.\n\n[5] Mitigating the alignment tax of rlhf.", "claims_and_evidence": "The theoretical claims are supported by proofs. The theory considers a simplified linear setting while the paper's overarching story is about large neural networks. The paper provides some empirical evidence with LLM benchmarks.", "methods_and_evaluation_criteria": "The choice of benchmarks is appropriate: MT-Bench (instruction-following), Commonsense-QA (general knowledge), and MMLU (multi-task understanding). The linear regression framework provides a controlled theoretical environment for studying the bias-variance trade-off.", "theoretical_claims": "I skimmed the statements and proofs and did not find any specific issues. I am not an expert here.", "experimental_designs_or_analyses": "I checked the soundness of the empirical evaluation setting.", "relation_to_broader_scientific_literature": "While the authors present these findings as novel (“surprising,” “intriguing”), similar phenomena—such as improved fine-tuning performance through model averaging—have already been observed and extensively studied in prior work. Section 3 reproduces the phenomenon on LLM benchmarks, which may be considered a contribution but isn't surprising.\n\nThe main novelty here lies in the theoretical contribution (Section 5), offering an explicit formalization of how ensemble methods manage the bias-variance trade-off within an over-parameterized linear setting.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "- The theory seems to be the main contribution of the paper. The experiments in section 3 feels disconnected from this core contribution. The theory could equally serve as a justification for any of the similar prior results on e.g. ImageNet models.\n- The paper provides a convincing theoretical justification for understanding why ensembling works.", "other_comments_or_suggestions": "- Figure 1 takes a lot of space but isn't informative.\n- In place of Figures 2 and 3, I'd suggest drawing the interpolation-style plots in e.g., Fig 1 of \"Robust fine-tuning of zero-shot models\n\". Currently it's unclear which ensembles are constructed from which fine-tuned checkpoints.", "questions_for_authors": "- Can your analysis be extended to make similar statements about (1) ensembles of multiple fine-tuned models or (2) weight-space ensembles? Such ensembles work well in practice (e.g., Model Soups); weight-space ensembles, in particular, are very useful because they don't increase parameter count/latency.\n- How does the overadaptation phenomenon relate to catastrophic forgetting? The paper mentions catastrophic forgetting but doesn't fully distinguish between overadaptation and catastrophic forgetting.\n- How does the sparsity assumption in Condition 2 relate to real-world fine-tuning scenarios? The theoretical setup assumes Task 2 (fine-tuning) has a \"sparse\" structure with many zero eigenvalues compared to Task 1 (pre-training). While this makes sense intuitively, is there a way you could measure this with LLMs?\n- Given that your theoretical analysis is based on linear models, which aspects of your findings (if any) do you think might not directly transfer to non-linear neural networks? Are there specific mechanisms in non-linear networks that might affect the benefits of ensembling?\n- How does your ensembling approach compare to parameter-efficient fine-tuning methods (e.g., LoRA, adapters) in terms of mitigating overadaptation? Could ensembling be combined with these approaches for additional benefits?\n- For LLMs, SFT is primarily an \"intermediate stage\" for post-training. Do you think your findings have valuable implications for RLHF/RLVR?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IMfY087qFJ", "reviewer": "Reviewer_faHj", "summary": "This paper focuses on theoretically and empirically studying task specific fine-tuning of large (over-parameterized) models.  It aims to provide theoretical underpinnings for empirical observations such as task specific fine-tuning leads to a performance degradation/forgetting on pre-training tasks, an early stopping or another form of regularization is needed to achieve good fine-tuning performance, and an ensemble of fine-tuned model with pre-trained model leads to a superior performance on fine-tuning data set as well as results in alleviated forgetting on pre-training tasks.  Authors present theoretical results, in a linear regression setting, on how regularizers are effective in reducing excess mean squared error on fine-tuning task and model ensemble improves it even further, and how regularization and model ensemble also provide improvement in excess mean squared error on pre-training task thereby leading to alleviation in forgetting.  These results are also interpreted in terms of bias-variance tradeoff and how the tradeoff is improved by regularization and model ensemble.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our sincere appreciation to the reviewer for all the constructive and insightful suggestions. Additional experiments and explanations have been provided to address the mentioned concerns.\n\n**Performance on pre-trained model, figure 2 and 3.** We have improved the figures accordingly and highlighted the checkpoint source of all ensembles in the figures (https://anonymous.4open.science/r/ICML2025-rebuttal-forgetting-1F68/README.md ). Additionally, interpolation weights of 0.0 and 1.0 are also included to cover the two extremes of the pure pre-trained and fine-tuned model respectively. Results show that the pre-trained model demonstrates moderate performance in commonsense reasoning, but shows poor performance in downstream tasks.\n\n**The connection between theoretical and empirical results.** Our experiment on LLM and theory match in showing the benefits of ensemble in fine-tuning. As fine-tuned parameters are close to pretrained points, we can adopt a linear setting with NTK explanation. Specifically, we can approximate the neural network $f(x, \\alpha)$ as $f(x, \\alpha_0) + \\nabla_\\alpha f(x, \\alpha_0)^T (\\alpha - \\alpha_0)$. Comparing this with our linear setting $y = x^T \\theta$, the features $\\nabla_\\alpha f(x, \\alpha_0)$ can be interpreted as the input x in the linear model, and the trainable parameter $\\alpha - \\alpha_0$ as $\\theta$. Since $f(x, \\alpha_0)$ remains unchanged during training, its effect can be disregarded in this simplification. Such simplifications can provide intuitive insights and have been widely used in prior works [1,2].\n\nRegarding our assumptions on x, the high-dimensional setting characterizes the “features” $\\nabla_\\alpha f(x, \\alpha_0)$ in overparameterized neural networks. We can further validate this assumption by analyzing the eigenvalue distribution of the Hessian matrix in practical models using PyHessian [3] and variants of Lanczos algorithms [4], which confirms the presence of several “large” eigenvalues and many “small” eigenvalues. \n\nAdditionally, the sparse structure observed in fine-tuning tasks reflects the nature of knowledge specialization across different inputs. While pretraining involves diverse inputs encompassing broad knowledge, fine-tuning is performed on specific tasks with a narrower scope, leading to a “sparse” structure in our theoretical formulation.\nThough bridging the gap between empirical observations and theoretical results still remains a challenging endeavor, this work represents a first step toward developing a theoretical framework for understanding this phenomenon. We aim to explore more comprehensive explanations in future research.\n\n**Explanation on Table 1.** Thank you for your insightful questions. For Llama-3-8B, Qwen2-7B, and Gemma-2-9B, we all train for 1 epoch. Regarding the ensemble methods, we adopt the weight of $1-\\tau=0.2$ for the pre-trained model, and $\\tau=0.8$ for the fine-tuned model. These experimental details will be included to later versions of the paper to improve clarity.\n\n[1] Mallinar, Neil, et al. \"Benign, tempered, or catastrophic: A taxonomy of overfitting.\" arXiv preprint arXiv:2207.06569 (2022).\n\n[2] Kumar, Ananya, et al. \"Fine-tuning can distort pretrained features and underperform out-of-distribution.\" arXiv preprint arXiv:2202.10054 (2022).\n\n[3] Yao, Zhewei, et al. \"Pyhessian: Neural networks through the lens of the hessian.\" 2020 IEEE international conference on big data (Big data). IEEE, 2020.\n\n[4] Zhang, Yushun, et al. \"Why transformers need adam: A hessian perspective.\" Advances in Neural Information Processing Systems 37 (2024): 131786-131823.", "claims_and_evidence": "I find the following claims to be weak:\n* Claim in Section 3 that empirical results with ensemble of fine-tuned and pre-trained model show magical performance of ensemble as compared to fine-tuned model only.  In these results it seems the baseline (epoch 0) performance to indicate how well the pre-trained model performs by itself on the tasks covered in the paper.  Similarly, when the fine-tuned model is interpolated with the pre-trained model, please include interpolation weights of 0.0 and 1.0 to cover those two extremes.\n* The claim that the theoretical analysis presented in the paper helps to explain the observed empirical results is also weak.  The theoretical analysis is conducted for a linear regression scenario with potentially strong assumptions on data distributions.  How that extends to the practical case of large language models and empirical data distributions is unclear.  The theoretical analysis is nice in itself (I did not validate its correctness, taking the results on their face value), so perhaps it should be presented by itself and not as a justification of empirical results with large models?", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Did not verify correctness of theoretical claims", "experimental_designs_or_analyses": "As mentioned above, experimental results are lacking in a couple of respects:\n* Baseline results that show how well the pre-trained model performs by itself on the tasks covered in the paper\n* When the fine-tuned model is interpolated with the pre-trained model to create the ensemble, interpolation weights of 0.0 and 1.0 are missing\n* Which model epoch(s) are used for Table 1 results?  And for the ensembles used in this table, what were the ensemble weights?", "relation_to_broader_scientific_literature": "The theoretical analysis and results on impact of model ensemble on excess mean square error are novel.", "essential_references_not_discussed": "None that I am aware of", "other_strengths_and_weaknesses": "Nothing beyond what’s covered in comments above", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i6mXI54OOm", "reviewer": "Reviewer_FtRM", "summary": "The paper theoretically analyzes why WiSE-FT or other regularization methods work. The authors also show that WiSE-FT is better than vanilla fine-tuning with regularization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for all the constructive suggestions.Additional experiments and explanations have been provided as follows.\n\n**The connection between theoretical and empirical results.** Our experiment on LLM and theory match in showing the benefits of ensemble in fine-tuning.As fine-tuned parameters are close to pretrained points, we can adopt a linear setting with NTK explanation. Specifically, we can approximate the neural network $f(x, \\alpha)$ as $f(x, \\alpha_0) + \\nabla_\\alpha f(x, \\alpha_0)^T (\\alpha - \\alpha_0)$. Comparing this with our linear setting $y = x^T \\theta$, the features $\\nabla_\\alpha f(x, \\alpha_0)$ can be interpreted as the input x in the linear model, and the trainable parameter $\\alpha - \\alpha_0$ as $\\theta$. Since $f(x, \\alpha_0)$ remains unchanged during training, its effect can be disregarded in this simplification. Such simplifications can provide intuitive insights and have been widely used in prior works [1,2].\n\nThe high-dimensional assumption on x characterizes the features $\\nabla_\\alpha f(x, \\alpha_0)$ in overparameterized models, i.e, there are several large eigenvalues and many small eigenvalues. It can be validated by the eigenvalue distribution of Hessian matrix in [3] [4].\n\nConsidering the loss function, to provide theoretical insights into empirical observations,we analyze the MSE loss, which intuitively captures the impact of bias and variance. From a practical standpoint, a well-performing model should exhibit a small gap between training and test loss.In such cases, the test prediction y’ is expected to be close to the training label y, given that the corresponding test input x' is similar to the training input x.Under this scenario, the behavior of MSE aligns with that of other loss functions,making this simplification reasonable and informative. Though bridging the gap between empirical results and theory still remains a challenging endeavor, this work represents a first step toward developing a theoretical framework for understanding this phenomenon. We aim to explore more comprehensive explanations in future works.\n\n**Conditions in Theorem 5.1.** In Theorem 5.1,to achieve good performance on both the fine-tuning and pretraining tasks, we prefer to set the regularization coefficient within $0 \\le \\lambda \\le \\lambda’$ and the ensemble coefficient within $ \\tau’(\\lambda) \\le \\tau \\le 1$. Generally, we have $\\lambda' = O(1/n)$, where n is the sample size. This suggests that the regularization penalty should not be too large, as excessive regularization may dominate the primary MSE loss. Meanwhile, $\\tau'(\\lambda)$ is a constant between 0 and 1 (often greater than 0.5), implying that the ensemble coefficient $\\tau$ should not be too small; otherwise, the information learned during fine-tuning may be lost. To validate these theoretical insights, we conduct several simulations (see Figures 4 and 5 in Appendix B). The results confirm that a small penalty coefficient $\\lambda$ and a large ensemble weight $\\tau$ consistently lead to improved model performance.\n\n**Performance on pre-trained model, figure 2 and 3.** We have improved the figures accordingly and highlighted the checkpoint source of all ensembles in the figures (https://anonymous.4open.science/r/ICML2025-rebuttal-forgetting-1F68/README.md ). Additionally, interpolation weights of 0.0 and 1.0 are also included to cover the two extremes of the pure pre-trained and fine-tuned model respectively. Results show that the pre-trained model demonstrates moderate performance in commonsense reasoning, but shows poor performance in downstream tasks.\n\n**Typos in L330, L363.** We appreciate the valuable feedback and will correct these terms in the main text.\n\n**Noisy training data in L370.** It refers to samples that contain noise. Let x denote the input and y denote the output, the relationship is $y = x^T \\theta + \\epsilon$, where $\\epsilon$ represents a noise variable. \n\n**Dolly.** Yes, Dolly is not used for training the Llama-3-8B base model. According to the [released huggingface model page](https://huggingface.co/meta-llama/Meta-Llama-3-8B), the llama-3-8B base model’s knowledge cutoff is March 2023, meaning the pre-training data utilized is until March 2023. Meanwhile, [Dolly](https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm) was released in 2023.4. Additionally, Meta explicitly states that their pretraining data comes from publicly available sources (https://ai.meta.com/blog/meta-llama-3/), so the possibility that they use Dolly from private sources is also excluded.\n\n[1] Benign, tempered, or catastrophic: A taxonomy of overfitting.\n\n[2] Fine-tuning can distort pretrained features and underperform out-of-distribution.\n\n[3] Pyhessian: Neural networks through the lens of the hessian.\n\n[4] Why transformers need adam: A hessian perspective.", "claims_and_evidence": "The claim is generally supported by theoretical analysis, but it is difficult to find reasons for showing LLM experiment results. The analysis itself is not designed specifically for LLM tasks. The LLM experiments make me question whether the theory is not applicable to the standard image classification task (Taori et al., 2020) used in previous studies the authors cited.\n\nSecondly, the authors use a linear regression task to show why ensemble methods (e.g., WiSE-FT). The authors need to justify why this simplification is valid (e.g., whether a proof holds for cross-entropy loss). I acknowledge that Kumar et al. (2022) also used a linear regression task, but that does not mean that this work does not need to be justified.\n\nFinally, Theorem 5.1 has conditions for each statement. I wonder how easily this condition is met in practice.", "methods_and_evaluation_criteria": "The authors chose reasonable benchmark and evaluation criteria for measuring performance drop in the robust fine-tuning of LLM.", "theoretical_claims": "I did not fully check the correctness of proofs in supplementary materials. Hence, my confidence score can be 1-2.", "experimental_designs_or_analyses": "The authors chose reasonable benchmarks and training for measuring performance drops in the robust fine-tuning of LLM. However, it is difficult to find the source of lines between ensemble methods in Figures 2 and 3. Considering the trajectories of WiSE-FT, it can have different shapes. The linear interpolation using five data points can show readers how ensemble results can be changed.", "relation_to_broader_scientific_literature": "The theoretical analysis can help understand robust fine-tuning.", "essential_references_not_discussed": "I do not have any concerns regarding the reference.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "It is good to show pre-trained model’s performance in each experiment.\n\nIn L 330, Regular. -> regularization or just Reg. Regular is confusing.\n\nIn L 363, Task1, because -> Task 1 because", "questions_for_authors": "Could you elaborate on what noisy training data in L370 is?\n\nCan you guarantee Dolly is not used for training the Llama 3 model? Considering both release dates, I suppose yes, but I'm not 100% sure.\n\n\n**[Post Rebuttal]** Thank you for the clarification. I keep my rating.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eZnBezFmAt", "reviewer": "Reviewer_PAsW", "summary": "The paper theoretically investigates the phenomenon observed in prior work that ensembling pre-trained and fine-tuned weights of foundation models outperforms dedicated fine-tuning strategies.\nThe authors first verify this phenomenon via empirical results for instruction fine-tuning LLMs.\nThen the authors provide a theoretical framework to compare the excess risk between different estimators that represent vanilla or regularized fine-tuning, as well as ensembles.\nIt is found that ensemble estimators attain a favorable balance between bias and variance error terms and hence improve performance while mitigating forgetting.\n\n## UPDATE\n\nThe authors have addressed all my concerns during the rebuttal period, hence I increased my score and recommend acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to extend our sincere appreciation to the reviewer for all the constructive and positive feedbacks on our contribution. Additional experiments and explanations have been provided to address the mentioned concerns.\n\n**Guidance for enhancing performance of fine-tuning strategies.** Some theoretical insights have been successfully applied to empirical experiments and provide guidelines for hyperparameter search. For example, the theory predicts that the value of $\\tau$ should be large, and the value of $\\lambda$ should be small, while in practice, the optimal $\\tau$ is in [0.4, 0.8], with the optimal $\\lambda$ in [1e-3, 1e-2]. This provides concrete empirical evidence and verifies the theoretical prediction.\n\n**The variance for the different fine-tuning methods on MT-Bench.** Additional multi-trial experiments on Norm-Penalty with Ensemble (https://anonymous.4open.science/r/ICML2025-rebuttal-forgetting-1F68/README.md ) show that the standard deviation on MT-Bench is ~0.06, which is sufficiently small for results in Table 1. Furthermore, an accuracy gap of 1.0 is generally considered non-trivial for MMLU and Commonsense-QA for large language models. For example, in the [llama-3.1’s post](https://ai.meta.com/blog/meta-llama-3-1/), llama-3.1-8B claims superiority over Gemma-2-9B-it with a MMLU accuracy of 73.0 v.s. 72.3.\n\n**More thorough empirical verification.** Thanks for the constructive suggestion! We have included further experimental results on LoRA (rank=32) (https://anonymous.4open.science/r/ICML2025-rebuttal-forgetting-1F68/README.md ) to justify the empirical benefits of the ensemble method, where consistent performance improvements have been observed across all benchmarks.\n\n**Relationship between early-stopping and ridge regularization.** Many thanks for the helpful feedback, we will add a remark in section 3.2 to elaborate the relationship between early-stopping and the ridge regularization.\n\n**Explanation for the usage of overadaptation and overfitting.** We apologize for the confusing usage of these two terms. In our work, \"overfitting\" and \"overadaptation\" refer to the same concept—training the model until the training loss becomes very small during the fine-tuning process. To maintain consistency, we will revise the main text and use the term \"overadaptation\" throughout.\n\n**Are the different points for ensemble in Figure 2 and 3 for different values of $\\tau$? If that is the case it would be good to mention that at least in the caption or have respective values for $\\tau$ at each dot of the ensemble curve.** In Figures 2 and 3 (https://anonymous.4open.science/r/ICML2025-rebuttal-forgetting-1F68/README.md ), the different points for ensemble models are from different $\\tau$ values and the fine-tuned models are trained with different seeds. We have modified Figure 2 and 3 to make the presentation more clear. \n\n**Are the different dots for fine-tune methods in Figure 2 and 3 for different seeds or different regularizers? Those markers should be labelled accordingly.** For the dots of fine-tune methods, we adopt both Norm-Penalty and DiffNorm-Penalty to fine-tune. We also use different seeds to repeat the experiments to provide more results. They are all plotted in Figures 2 and 3 (https://anonymous.4open.science/r/ICML2025-rebuttal-forgetting-1F68/README.md ).", "claims_and_evidence": "The authors say that their theoretical results provide guidance for enhancing the performance of fine-tuning strategies. I appreciate the empircal validation of Theorem 5.1, but this does not resemble a real-world fine-tuning setting as in Section 3. Can the authors be a bit more specific what the claimed guidance for enhancing performance of fine-tuning strategies is?", "methods_and_evaluation_criteria": "The methods employed for empirical evaluation and evaluation criteria are sound.", "theoretical_claims": "The main merit of this work is of theoretical nature.", "experimental_designs_or_analyses": "What is the variance for the different fine-tuning methods on MT-Bench? The difference in the scores seem marginal, are they significant?\nFurthermore, does the same trend persist for the remaining two evaluation sets?\nIt is important to establish whether this phenomenon persists across several datases to determine its significancy.", "relation_to_broader_scientific_literature": "The paper positions itself by analysing a practical phenomenon observed during forming ensembles for fine-tuning of large models, which has been observed in prior work.\nHence, the major contribution of the work is of theoretical nature, which is valid and relevant to foster theoretical understanding.", "essential_references_not_discussed": "To the best of my knowledge the most relevant works have been discussed.", "other_strengths_and_weaknesses": "**Strengths**\n\nThe work is well motivated and well-written for the most part.\n\nThe theoretcal contrbution seems rigorous and I enjoyed the empirical validation of Theorem 5.1.\n\n**Weaknesses**\n\nOne weakness in my view is the lack of guidance for fine-tuning strategies (see Claims and Evidence).\n\nAnother one is a more thorough empirical verification of the observed phenomenon.", "other_comments_or_suggestions": "The relation between early-stopping and ridge regularization was not clear to me from the empirical sections. The first time this connection is established and backed with references is in line 270 (right), to enhance clarity, I suggest elaborating no this relation earlier in the empirical part which is also serves as justification for the choice of regularizer.", "questions_for_authors": "- Is there any particular reason why the authors jump back and forth between the terms \"overadaptation\" and \"overfitting\"? It would be easier to follow if one term would be used consistently.\n - Are the different points for ensemble in Figure 2 and 3 for different values of $\\tau$? If that is the case it would be good to mention that at least in the caption or have respective values for $\\tau$ at each dot of the ensemble curve.\n - Are the different dots for fine-tune methods in Figure 2 and 3 for different seeds or different regularizers? Those markers should be labelled accordingly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper explores the effectiveness of ensembling pretrained and fine-tuned language models to combat the issue of catastrophic forgetting in SFT. The authors demonstrate that ensembling helps retain general knowledge in language models. More notably, they uncover an overadaptation effect where the ensemble model not only preserves general capabilities but also outperforms the fine-tuned model on its own target domain. To explain this, the paper presents a theoretical analysis centered on the bias-variance trade-off, showing that ensembling strikes a more effective balance than traditional regularization techniques. The analysis, grounded in overparameterized linear models, is supported by empirical results that highlight the benefits of interpolating between pretrained and fine-tuned weights.

The paper received a positive recommendation for acceptance. One reviewer praised the significance and novelty of the findings, emphasizing their relevance to the broader community. A critical point raised by a negative reviewer was addressed in the authors’ rebuttal, which clarified the misunderstanding. The paper demonstrates known behaviour in vision for language models, which may not be as significant.  In the final recommendation, I support acceptance, with a suggestion that the authors incorporate these clarifications, including issues raised by the reviewers, into the camera-ready version to ensure clarity and completeness.
Decision: Accept (poster)