LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Authors: Xinyue Zeng
Haohui Wang
Junhong Lin
Jun Wu
Tyler Cody
Dawei Zhou
Venue: ICML 2025 poster (2025)
Keywords: LLM Selection
PAC-Bayesian Theory
Generalization Bound
Scaling Laws
Primary area: deep_learning->large_language_models
TL;DR: We derive a PAC-Bayesian generalization bound for LLM fine-tuning dynamics and propose LENSLLM, a framework that enables accurate, efficient performance prediction across diverse tasks.

Abstract: The proliferation of open-sourced Large Language Models (LLMs) and diverse downstream tasks necessitates efficient model selection, given the impracticality of fine-tuning all candidates due to computational constraints. Despite the recent advances in LLM selection, a fundamental research question largely remains nascent: *how can we model the dynamic behaviors of LLMs during fine-tuning, thereby enhancing our understanding of their generalization performance across diverse downstream tasks?* In this work, we propose a novel theoretical framework that provides a proper lens to assess the generalization capabilities of LLMs, thereby enabling accurate and efficient LLM selection for downstream applications. In particular, we first derive a *PAC-Bayesian Generalization Bound* that unveils fine-tuning dynamics of LLMs and then introduce *LensLLM*, a Neural Tangent Kernel (NTK)-based Rectified Scaling Model that enables accurate performance predictions across diverse tasks while maintaining computational efficiency. Extensive empirical results on 3 large-scale benchmarks demonstrate that our model achieves up to 91.1% accuracy and reduces up to 88.5% computational cost in LLM selection, outperforming 5 state-of-the-art methods. We open-source our proposed *LensLLM* model and corresponding results at [LensLLM.io](https://github.com/Susan571/LENSLLM.git).

LENSLLM:
Unveiling Fine- Munnng Dynamics for LLM Selection
Xinyue Zeng' Haohui Wang' Junhong Lin
Jun Wu
Tyler Cody
Dawei Zhou'
Abstract
ModetSize NLPmotries
ZeroShoe Rectified Scaling Law
Sab Tunlunnnn
Le (Ouns)
The proliferation of open-sourced Large Lan- guage Models (LLMs) and diverse downstream tasks necessitates efficient model selection, given the impracticality of fine-tuning all candidates due to computational constraints. Despite the recent advances in LLM selection, a fundamental re- search question largely remains nascent: how can we model the dynamic behaviors of LLMs during fine-tuning, thereby enhancing our understand- ing of their generalization performance across diverse downstream tasks? In this work, we pro- pose novel theoretical framework that provides a proper lens to assess the generalization capabili- ties LLMs, thereby enabling accurate and effi- cient LLM selection for downstream applications. In particular, we first derive PAC- PAC-Bayesian Gen- eralization Bound that unveils fine-tuning dynam- ies of LLMs and then introduce LENSLLM, Neural Tangent Kernel (NTK)-based Rectified Scaling Model that enables accurate performance predictions across diverse tasks while maintain- ing computational efficiency. Extensive empirical results on 3 large-scale benchmarks demonstrate that model achieves up 91.1 % accuracy and reduces up to 88.5% computational cost in LLM selection, outperforming 5 state-of-the-art meth- ods. We open-source our proposed LENSLLM model and corresponding results at LensLLM.io.
100
77.9
80.1
83.5
85.8
91.1
89.2
90
66.8
83.3
84.4
PeurCorr
20
80 8
77.3
-20
27.4
-60-
-65
Giguword
Figure I. Our model demonstrates superior performance on Giga- word (See al., 2017), achieving Pearson Correlation Coefficient of to 85.8% and Relative Accuracy of up to 91.1%, surpass- ing 5 state-of ofte-oofthheart the-art methods for LLM selection. (Higher values indicate better performance)
progress in applications such machine translation (Liu et al., 2020; Team et al., 2022), text summarization (Zhang et al., 2020; Lewis et al., 2019), question answering (Khashabi et al., 2020; Wei et al., 2022c), and dialogue systems (Bommasani et al., 2021; Thoppilan et al., 2022). However, the recent explosion open-source LLMs (c.g.. LLaMA (Touvron al., 2023), Falcon (Almazrouei et al., 2023), Mistral (Mistral AI, 2024), DeepSeek (DeepSeek- AI, 2024)) has heightened interest in model selection. This selection aims to identify the optimal model from set of candidates by balancing model complexity with the ability to explain the observed data. The tremendous variety of downstream tasks (Wei et al., 2022b; Wu et al., 2023; Wang & Duan, 2024; Jiao al., 2023) and the exponential growth in model sizes (Zhao al., 2024; Minaee al., 2024), as well the associated computational costs, have led the need to select the optimal LLM efficiently. Moreover, the models selected through fine-tuning historical data or spe- cific tasks may fail in novel scenarios or -isiiiibbiittnonnn cases (Wei et al., 2022b; Schaeffer et al., 2023), which also raises the urgent need for a generalized model selection framework for LLMs.
1. Introduction
The rise of large language models (LLMs) has revolu- tionized natural language processing, driving remarkable
'Deparmment of Computer Science, Virginia Tech, Blacksburg. VA, USA. 'Department of Electrical Engincering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA. 'Department of Computer Science and Engineering, Michi- gan State University, East Lansing. MI, USA. 'Department of In- telligent Systems Division, Virginia Tech, Blacksburg, VA, USA.. Correspondence to: Xinyue Zeng <xyzeng @tt.enngvt..du>.... vt.edu>..
Model selection is Long-standing research problem in the machine-learning community. Traditional model selection approaches (Yang et al., 2023), designed for small-scale ma- chine learning models, are inadequate for LLMs due to poor generalization and prohibitive computational costs. More recently, the machine learning community has observed in-
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
creasing attention on characterizing the unique behaviors of modern LLMS and developing tailored selection meth-
Table Notations.
Symbol
Description
ods (Haowei et al., 2024). However, these methods remain mostly heuristic, often depending on strong assumptions or iterative trial-and-error processes. These limitations under-
X
Input feature space
D
Probability distribution of samples
score the necessity for a theoretical grounding for LLM se- lection that can elucidate fine-tuning dynamics and achieve better generalization in various downstream applications.
M
Space of LLMs
Sample size
I
Depth of neural architecture (numher of layers) Expected loss on the test distribution
Despite the importance of LLM selection, there are two
L
Empirical loss computed on test set Weight difference vector at layer Feature vector from Targeted label of
fundamental challenges that largely remain nascent. C1. Theoretical Understanding: There is limited theoretical analysis characterizing the dynamic behaviors in the LLM fine-tuning process, particularly the pre-power phase that emerges in low-data regimes. A rigorous theoretical foun- dation is crucial for providing proper lens in examining LLM selection and achieving model generalization. C2. Ac- curate and efficient LLM Selection Framework: There always exists tension between model selection accuracy and computation cost. How can we find an optimal trade- off framework that could accurately select models across various tasks in resource-constrained scenarios?
y
W
Pre-trained weight matrix layer
W
Fine-tuned weight matrix at layer Non-negative truncated Hessian matrix
H
tion (e.g., 2)), boldface lowercase letters to denote vectors (e.g., v,), and boldface uppercase letters to denote matrices (e.g.. W.).
2.1. Scaling Laws in LLM Fine-Tuning
In this work, we present three key contributions that ad- vance the field of LLM selection. First, we develop PAC- Bayesian Generalization Bound that reveals distinct pre- power and power phases in fine-tuning. This theoretical foundation provides a novel framework for examining LLM selection, enabling robust model generalization across di- verse tasks. Second, building on our theoretical results, we introduce LENsLLM,, which leverages NTK (Jacot al., 2018) approximate the LLM fine-tuning scaling laws. By modeling the transition between pre-power and power phases, our approach not only achieves accurate LLM se- lection across tasks but also reduces computational costs by more than 50% compared to our best competitor Sub- Tuning (Kaplun et al., 2023). Finally, our comprehensive empirical evaluation demonstrates the superior performance of our model. For representative example, Figure shows that our model achieves up to 91.1% relative accuracy and 85.8% Pearson correlation. Moreover, it reduces computa- tional costs by up to 88.5% compared to FullTuning while maintaining comparable performance. We open-source our proposed LENSLLM model and corresponding results at LensLLM.io.
With the rise of L.Ms,, researchers in the machine-learning community have increasingly focused on understanding and characterizing the behavior of these models during the fine- tuning process. A key area of investigation is the scaling laws governing model performance, which has been exten- sively studied for the pre-training stage (Henighan et al., 2020; Kaplan et al., 2020; Bahri et al., 2024). These stud- ies propose that model performance follows predictable power-law relationship, described below:
Definition 1 (Power-law in Kaplan et al. (2020)). The scal- ing loss L(s,-) is a function of model size N and training set size D, i...
A Nax
B D
a
L(N,D)
+
(I)
Here a. are universal parameters to be fitted. This groundbreaking work revealed that model performance consistently improves a power law with increases in three critical factors: model size, training data, and computational power. These findings laid a strong foundation for under- standing model behavior during pre-training.
2. Preliminary
Building on this, Haowei et al. (2024) demonstrated that fine- tuning performance depends not only on model size N but also on various architectural design choices, including the number of layers, attention heads, and hidden dimensions. This intricate dependency complicates model selection using traditional scaling laws. However, to predict performance for specific models, a simplified version of the scaling law can be employed by excluding architectural considerations. Scaling laws for fixed models, as proposed in (Kaplan et al.,
In this section, we begin with the introduction of notations in Table 1, followed by scaling law behavior in LLM fine- tuning, generalization bound, and our problem definition. We consider the fine-tuning of LLMs in a supervised learn- ing setting. The key notations used throughout this paper are summarized in Table 1. We use regular letters denote scalars (e.g., !), italics letters to denote space and distribu-
2

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
2020; Hernandez et al., 2021; Tay al., 2022), exhibit the following form:
further data collection against the expected performance gains.
B DJ E
2.2. Generalization Bound
L(D)=
While empirical studies offer valuable insights, theoretical frameworks are equally essential for understanding the dy- namics of fine-tuning, Among these, generalization bound stands out as powerful tool, providing rigorous mathemat- ical insights into fine-tuning performance. Prior work by Ju et al. (2022) introduced a generalization bound tailored to fine-tuned feed-forward neural networks, serving as a pre- liminary lens for analyzing fine-tuning dynamics. However, their analysis does explicitly account for transformer- specific architectural elements, which limits its applicability to modern large-scale language models.
where D represents the training set size, while B, E, a, and 33 are parameters specific to the model and task.
As illustrated in Figure 2, two distinct phases emerge in the scaling behavior of fine-tuning test loss (L) with training sample size (D): the pre-power phase and the power phase. The pre-power phase occurs in low-data regimes where model behavior is dominated by initialization and early train- ing dynamics. As the training size increases and reaches a transition point, models enter the power phase, characterized by predictable scaling behavior. In this phase, the relation- ship between training size and test loss follows a nearly lin- car correlation, as widely studied in prior works (Henighan et al., 2020; Kaplan et al., 2020; Bahri et al., 2024).
Theorem 1 (Generalization Bound in et al. (2022)), As- sume the activation functions (-) for i i l,.... and the loss function L are all twice- siice-iifferentable and their first-order and second-order derivatives are all Lipschitz- continuous. Suppose [[f y) (L(fw) used in the fol- lowing for simplicity) is bounded by fixed value C for any œ E X with class label y. Given an l-layer network fw. with probability at least 0.99 (or equivalently, almost everywhere in the statistical sense), for any fixed e close to zero, we have
for
LaMini-124M BART Base GPT2
(L)
Pre-pomer
VH L(W) s (1+0)(())) (1 (1+0)0 VH
+ (3)
Vn
where Hi is defined as (aX() U, H [[W]), for all i = I,. and = O(n 3/4) represents an error term from the Taylor's 's expansion.
Power Phase
100
10a
30
10
10f
Train Size (D)
2.3. Problem Definition
Figure 2. Phase transition in fine-tuning test loss (L) scaling with training sample size (D). The data reveals pre-power phase at small D, followed by the established power phase showing the linear correlation between L and D.
In the context of LLM selection, we aim to identify the optimal model from a set of candidates for specific down- stream tasks under resource-constrained scenarios. Without loss of generalization, we denote S as training dataset from DD and M {m,, m2, ..., my) } as set of candidate mod- els. For each model m,, there is associated with feature vector representing its characteristics (e.g.. model size, architecture, training data).
Understanding the underlying mechanism of this phase tran- sition phenomenon is crucial for effective model selection. While prior work has empirically observed the emergence of power-law behavior during fine-tuning, it has largely lacked a theoretical framework to explain when and why this transition occurs. This theoretical gap limits the ability to make informed decisions about data efficiency and model scaling. In contrast, our theoretical analysis provides a first- principled understanding of the transition dynamics. First, it enables precise predictions about when additional train- ing data will trigger the onset of the power phase, resulting in consistent and predictable performance improvements. Second, once a model enters the power phase, our scaling framework offers guidance on how to balance the costs of
Given the notations above, we formally define the problem as follows:
Problem LLM Selection in Resource-Constrained See- narios
Given: (1) Limited training data S; (2) A set of candidate LLMs M = {mi, Ma,,,,,,,,.,,,,. with their corresponding feature vectors x.
Objective: The optimal model m² E -M on S that has the best performance.
3

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
3. Model
formers and lead to our following main theoretical founda- tion:
In this section, we introduce our framework, LENSLLM, for analyzing LLM selection. The core idea lies in modeling and regularizing the dynamics of transition phases observed during fine-tuning. Specifically, we begin by deriving novel PAC-Bayesian Generalization Bound for these tran- sition phases, incorporating the role of the Hessian matrix to capture fine-tuning behavior. Building on this theoreti- cal foundation, we develop the overall learning paradigm of LENSLLL,, focusing on effectively characterizing model be- havior both the pre-power and power phases. Finally, we present an optimization algorithm for LENSLLM, including detailed pseudo-code to illustrate its implementation.
Theorem 2 (PAC-Bayesioe Generalization Bound). For
fine-tuned transformer model fa satisfying Assumptions 1-3, with probability at least 0.99 and any fixed > 0:
Lifa) - (c+ (e+ (1 +ONCE Vr, +0(n
(4)
vn
where h4 2 maX(A of H (ff)))o.
This bound provides a theoretical foundation for character- izing the fine-tuning dynamics of transformers and serves as a basis for analyzing the transition phases. The proof for this theorem is provided in Appendix A.
3.1. Theoretical Analysis on Fine-tuning Scaling Law
Understanding how language models behave during fine- tuning is essential for making informed decisions about model selection. While we have learned from empirical studies, we still lack theoretical framework to explain the mechanism of transition between different phases in fine- tuning. This understanding could help optimize the trade-off' between computational costs and model performance. To this end, we aim to use our proposed PAC-Bayesian Gener- alization Bound to explain pre-power and power phases in fine-tuning.
Understanding the underlying mechanism of phase transi- tions in fine-tuning is essential for LLM selection. While empirical studies provide valuable insights, there is still lack of theoretical foundations to reveal the dynamics of transition phases. Here we provide a theoretical foundation, PAC-Bayesian Generulization Bound, for understanding dy- namic transition phases during LLM fine-tuning, This leads to a more nuanced analysis of fine-tuning dynamies, par- ticularly pre-power and power phases in scaling laws. Our analysis is built upon the following key assumptions that account for sranefi properties:
First, we start from the following property of the Hessian- related term h4:
Assumption (Smoothness). The activation functions di on layer and loss function L are wrice-differemtiabee with Lipschitz-continee continuostti first and second derivatives (Allen-Zhu et al., 2019):
Proposition 1. Let {hi)i=1 be sequence
of Hessiaan-related values that satisfy h, (a.))))) of H [L(s))oo
Løllx- -ll,, where L& is the Lipschitz constant that bounds how fast the gradient of th can change;
(I) [V) - Voili)l
(1) By applying Cauchy-Sehwar theorem (Steele, 2004), we have the following inequality:
(2) (+))(()) V'ox(t)) S
la yll. where Las is the Lipschitz constant for the second derivative of dz.
ΣVh.
h4
(5)
1=1
Assumption 2 (Boundedness). Following Wei et al. (2019), the loss function and input features satisfy uniform bounds:
(2) Based on the definition of h, we have the following upper bound:
L. & C and ||||| < B.
h4 S C2n -2 C2n-92
(6)
where C2 is independent of n and B2 $ 3,.
Assumption 3 (Transformer Stability). Based on Liu al.
(2022), transformer components satisfy:
Leveraging the above key properties, we derive the follow- ing corollary I to further analyze the fine-tuning dynamics. Detailed proof has been provided in Appendix B.
(1) Attention mechanisms are LA-Lipschitz continuous (Lee et al., 2020);
(2) Attention scores are bounded: [|softmax(QKi QKT) S
Corollary 1. For any > 0, with probability over 0.99, under Assumptions 1-3, considering the properties of the Hessian matrix, the PAC-Bayesian Generalization Bound could be extended to:
M (Dong et al., 2021);
(3) Residual connections maintain gradient flow:
||V f=)]] 2 0>
(1++ + Cyn + +O(n (7)
(4) Layer normalization preserves scale: [LN =
[x|]] ||||2-
where C3 = VC and Ba = 8.+1 2
are both
These assumptions enable us to extend Theorem to trans-
modeltask-depenten..

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Remark (Pre-power Phase): The model's performance improves slowly during initial fine-tuning, with generaliza- tion errоr decreasing at rate O This phase is marked by high Hessian values and significant parameter sensitivity, necessitating careful tuning and substantial data for reliable adaptation.
the learning difficulty, B adjusts the initial test loss and E denotes the optimal loss of the model given infinite amount of data.
The design of this loss function is intentional and serves two key purposes: (1) It models the competing effects between the transformer's intrinsic learning dynamics (captured by NTK) and the dataset size: (2) It naturally incorporates pre-trained knowledge through the initial state (X).. In particular, the strategic placement of F(O, t) in the denom- inator represents significant advancement over traditional rectified scaling laws. This positioning explicitly accounts for pre-trained data influence through the NTK term, while the additive relationship between F F( t) and D creates unified framework that simultaneously captures both pre- training effects and fine-tuning data scaling.
Remark 2 (Power Phase): As ri increases, the error scaling transitions to C3n fs, becoming the dominant term. This phase demonstrates reduced Hessian values and enhanced stability, enabling more aggressive parameter updates and improved data efficiency.
Remark 3 (Phase Transition): The transition from the pre- power phase to the power phase is marked by the change the dominant constant factors, from O( - 4) to Can 9s which reflects the change in the Hessian values and parame- ter sensitivity.
3.2. LENSLLM Algorithm for LLM Selection
Algorithm 1 LENSLLM Algorithm
Input:
Building upon our previous theoretical analysis of LLM fine- tuning through the lens of our PAC-Bayesiaa Generalization Bound, now introduce LENSLLM, novel -Tsed framework for model selection. Our approach provides bridge between our theoretical analysis and practical model selection by leveraging the NTK to transformers so as to capture fine-tuning dynamics.
Training subset S with size D, models from space M, Regression threshold y, Stop threshold T
Output:
Predicted performance score r on the full dataset: r exp(V(log DI)) and minimal proportion of training data needed to reach the Pareto-Optimality curve s
2a
To formalize this connection, we first define an NTK matrix for transformer model -(:-)) with parameters @ at step t:
Initialize collection C= O for (training size, test loss)
pairs and
for iteration steps.
2: while TRUE do
(z x²) (a 0())- Vaf( (())
(8)
3: Train m E .M on S and obtain NTK matrix O and training steps / and corresponding loss L.
where x and x' represent the feature vectors of pre-training and fine-tuning, respectively. These features encapsulate relevant properties of the input samples that influence the fine-tuning process. To simplify, we use O to present the NTK matrix in the following.
4: if [C] 2 y then
Train regression estimator &b on C
Calculate deviation set A {I log - (log D)I} for all pairs in C and the standard deviation of the fitting residuals
To further characterize the mechanism of transition phases in fine-tuning, we define the following NTK-based test loss function on transformers:
7:
if s = llg L - (log D)1/ > 7 then
8:
break end if
9:
10:
end
11: Add (log IP). L) C
F(O,t) t) = (fo(( - -y)
12: Randomly select samples from S with 2 as S
13:
Update S S"
where 7j is the learning rate, represents the early stopping time in training steps during fine-tuning, fo( denotes initial outputs, and y represents true labels.
14:
Update +- a +
15:
end while
Motivated by these theoretical insights and their alignment with our theoretical analysis, we propose LENSLLM, Hessian-aware rectified scaling model:
Algorithm 1 presents a two-phase performance prediction method for LLM selection through iterative dataset reduc- tion. Starting with empty collection C, the algorithm trains an LLM on progressively smaller datasets and records size-loss pairs. For each iteration, it trains the model to obtain test loss Σ, NTK matrix θ, and training steps t. Af- ter collecting 7 pairs, it fits a regression estimator & and computes stopping signal based on the deviation from
B L(D) = F(O, + Dø
+
(10)
where F(O,t) t) is the adapted NTK-basee based test loss function on transformer, D is the number of training data, denotes
5

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
predicted values. The process continues until the deviation exceeds threshold r. The optimization process of our model in experiments consists of two phases: (I) a phase-fitting phase where parameter t is fixed for each dataset size while optimizing B, (3, and E to fit the observed test loss, and (2) a test loss prediction phase where all parameters (t, B, 3, and E) are jointly optimized to predict test loss across dif- ferent dataset sizes and training durations. In each iteration, the dataset size is halved through random sampling until termination.
and LaMini-774M. This broad selection captures variabil- ity across model sizes, training objectives, and architecture types, providing a rigorous basis for evaluating the effec- tiveness and stability of our fine-tuning approach.
Optimization: All models are fine-tuned using the A optimizer with a weight decay of 0.01, and all experiments are conducted on single NVIDIA A100 GPU with 80GB of memory. To characterize the fine-tuning dynamics across different model architectures and data sizes, we estimate B, E, B, L for each model by minimizing the loss function:
In the implementation, given a limited training dataset S and a set of candidate LLMs M, we aim to select the optimal model m E .M with the largest r.
min E. [LSE( log B - log(((()))))) + Df)) log B) - log I.(Da)
B.E e
Where L(D:) denotes the test loss of fine-tuning on the data size D,. and LSE denotes the log-exp-sum operator, used for numerical stability in modeling loss scales. This optimiza- tion allows us to capture the relationship between model capacity, data size, and fine-tuning efficacy in a unified, interpretable framework.
4. Experiments
In this section, we evaluate LENSLLM's performance LLM selection through two aspects: (a) evaluating its ef- fectiveness against baseline models, including fitting the transition phases in fine-tuning, predicting test loss on ad- ditional training data and selecting the optimal model; (b) evaluating its efficiency by calculating computational costs. Results consistently demonstrate our model's capability in accurate and efficient LLM selection.
Evaluation Matries: We use two established metries from Haowei et al. (2024) to evaluate the performance of meth- ods LLM selection: Pearson correlation coefficient (PearCorr) and Relative Accuracy (RelAcc). PearCorr mea- sures the correlation between selection scores and actual fine-tuning performance, evaluating a method's ability to rank models effectively. RelAce is defined as the perfor- mance gap between the selected model and the best model normalized by the gap between the worst and best models.
4.1. Experimental Setup
Benchmark: For robust evaluation across various tasks, we experiment with three benchmark datasets: FLAN (Wei et al., 2022a), Wikitext (Merity et al., 2016), and Giga- word (See et al., 2017). All of them are open-sourced on Hugging Face To analyze performance scaling, we create smaller datasets by randomly sampling examples ranging from 200 to 1,638,400 400 (doubling at each step), then fine-tune and evaluate models on a separate test set.
4.2. Effectiveness Analysis on LLM selection
We evaluate LENSLLM's effectiveness for LLM selection through three analyses: (1) comparing its curve-fitiing accu- racy during fine-tuning against Rectified Scaling Law (the previous state-of of-the-art method), (2) measuring its predic- tive performance using Root Mean Squared Error (RMSE) between predicted and actual test losses against Rectified Scaling Law, and (3) benchmarking LLM selection per- formance against five baseline methods (including Recti- fied Scaling Law) across FLAN, Wikitext, and Gigaword datasets using evaluation metrics.
Baselines: We evaluate our approach against five estab- lished baseline methods: (1) Rectified Scaling Law uses pre- trained- cained-datt-adaann adaate scaling law model to select (Haowei et al., 2024), (2) NLP metrics uses proxy generalization met- rics for selection (Yang et al., 2023), (3) SubTuning uses the performance of subset fine-tuning tine-tunig as selection score (Kaplun et al., 2023), (4) ModelSize uses logarithm of the number of model parameters for selection (Villalobos et al., 2022), and (5) ZeroShot uses zero-shot performance as selection criteria (Kojima et al., 2022).
We evaluate the curve- fittinn accuracy ca- pacity during fine-tuning. As shown in Figure 3, our model (blue square) consistently outperforms Rectified Scaling Law (red triangle) in predicting performance across archi- tectures. For OPT-1.3b, our model provides smoother and more accurate predictions that closely track the actual test loss curve, while Rectified Scaling Law shows notable fluc- tuations. Our model's superiority is particularly evident in GPT-2's middle to late training stages, maintaining steady alignment with ground truth compared to Rectified Scaling Law's 's volatile predictions. Similarly, for T5-base, our model demonstrates more stable predictions throughout training,
Curve Fitting:
Models: To ensure the generality and robustness of our find- ings, we conduct comprehensive fine-tuning experiments across a diverse suite of pre-trained language models span- ning a wide range of architectures and parameter scales. Specifically, our evaluation covers seven model families, including OPT-350M, OPT-1,3B, OPT-6.7B, T5-Small, T5- Base, Cerebras-256M, Cerebras-1.3B. mT5-Base, mT5- Large, BART-Base, BART-Large, GPT-2, LaMini- 124M
6

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Original
Roctifie Modal
Le (Ours)
RMSC Band (Reviffed)
RMSE Band (Ours)
OPT-1.3B
GPT-2
T5-Base
4.5
Yed (
2.5
4.5
FLAN
233
5.9
2.4
2.1
3.7
2.
35
1.9
2.0
3.1
299
1.8
22
2.8
1.9
HMSS: (laccieed 0.114
HASSE (Ptec U.3
Wikitest
2.0
HMSE (Duns) B:C34
26
RAISE (Oun) 0.609
1.
Yea Less
2,4
1.5
2.2
2.0
18
1.1
3.
1.4
28
55
Gigaword
Teat yos (2)
26
311
28
1.2
(Cum))
24
27
22
25
1.0
25
21
1.9
0.8
1.7
1.4
1.5
0.8
10"
16
10
10
10'
10"
Trininn Size (D)
Triinin Size (D)
Training Size (D)
Figure 3. Performance comparison showing the superior effectiveness LENSLLM (our method) across OPT-1.3b, GPT-2, and TS-base architectures on FLAN, Wikitext, and Gigaword datasets. LENSLLM consistently achieves significantly lower RMSE values (shown in a blue square) compared to the Rectified Scaling Law (shown a red triangle), with notably smaller error bands indicating more stable performance.
effectively capturing the gradual loss descent without the oscillations seen in Rectified Scaling Law.
achieves significantly lower errors across all architectures: Wikitext, errors are typically 5 times smaller (e.g.. OPT- 6.7B: 0.026 vs 0.132, mT5-Large- 0.028 vs 0.144); on FLAN, we maintain low RMSE (0.022-0.035) compared to Rectified Scaling Law's 's higher range (0.087-0.15)) and on Gigaword, our model shows consistent performance below 0.036 while Rectified Scaling Law varies between 0.094- 0.146. These results across three datasets and fourteen archi- tectures confirm our model's superior accuracy in predicting training dynamics.
Table 2. RMSE comparison between predicted and actual test
losses (x10-)) ) of our model and Rectified Scaling Law.
Wikitext
FLAN
Gigaword
Model
Ours
Rect
Ours
Rect
Ours
Rect
OPT-350M
0.2
1.10
0.32
1.50
0.26
0.98
OPT-1.3B
0.32
1.14
0.32
1.20
0.28
0.99
OPT-6.7B
0.26
1.32
0.26
1.31
0.26
1.46
T5-Small
0.35
1.01
0.28
1.30
0.3
1.27
LLM Selection: We compare our approach against five Rectified Scaling Law, NLPmetrics, Sub- Tuning, ModelSize, and ZeroShot. All the performance tested on a held-out validation set. Table presents compre- hensive model selection results across FLAN, Wikitext, and Gigaword datasets, demonstrating our method's exceptional performance. Our approach consistently achieves superior correlation (reaching 85.8% PearCorr) between predicted and actual model performance, significantly outperforming established baselines such as Rectified Scaling Law and NLPmetries. The robust performance further validated by outstanding RelAce scores (up to 91.1%), indicating models selected by LENSLLM consistently approach op- timal performance levels across all tasks. This substantial improvement over baseline methods establishes our frame-
T5-Base
0.32
1.30
0.26
1.26
0.3
0.94
baseline methods:
Cerebras-256N.
0.24
1.27
0.22
1.1
0.33
1.30
Cerebras-L.33 1.3B
0.26
1.18
0.32
1.00
0.28
1.00
mT5-Base
0.26
1.17
0.32
1.22
0.17
1.07
mTS-Large
0.28
1.44
0.32
1.07
0.28
1.10
BART-Base
0.3
1.27
0.3
0.96
0.26
0.99
BART-Large
0.17
1.31
0.28
0.87
0.36
1.14
GPT-2
0.3
1.30
0.3
1.23
0.26
1.33
LaMini-12444
0.28
1.01
0.35
1.00
0.3
1.15
LaMini-774M
0.32
1.14
0.28
1.13
0.28
1.19
We evaluate the test loss predic- tion performance of LENSLLM. Table 2 demonstrates our model's superior performance through RMSE compar- isons between predicted and actual test losses. Our model
Test Loss Prediction:
7

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Table Model selection results (PearCorr, RelAcc) of model (I.NSSILLM) and haselines (Rectified Scaling Law, NIPmmetrsss
SubTuning, ZeroShot and ModelSize) on three datasets in percentage. The best result within the same dataset is in bold font, and the second best result underlined.
Dataset
Metric
LENSLLM
Rectified Scaling Law
NLPmetrics
SubTuning
ZeroShot
ModelSize
PearCorr RelAcc
78.4
75.2
76.1
70.5
-12.8
-29.8
FLAN
88.3
85.7
84.7
80.4
82.3
75.6
PearCorr
82.6
80.3
78.9
75.4
6.1
38.0
Wikitext
RelAcc
90.2
88.8
87.5
85.1
84.4
68.5
PearCorr
85,8
83.5
80.1
77.9
-53.2
-27.4
Gigaword
RelAce
91.1
89.2
86.8
84.4
83.3
77.3
work as a reliable guide for practitioners in optimal model selection across diverse application scenarios.
comparison, we account for both the number of training epochs and hyper-paraman parameter search rounds h when cal- culating total computational costs across different tuning approaches. The performance is evaluated by the number of FLOPs when PearCorr of methods reach the curve (as illustrated in Figure 4).
4.3. Efficiency Analysis LLM Selection
1.0
As shown in Table 4, LENSLLM achieves superior computa- tional efficiency through its innovative progressive sampling strategy: it reduces computational costs by up to 88.5% compared to FullTuning while maintaining comparable per- formance. LENSLLM achieves computational costs of 0.48, 0.59, and 0.97 x x1021 FLOPs across tasks, which substan- tially outperforms both SubTuning and Full'unnig.
0.8
Acunayy Modian Con
PearCorf
0.6
Midler
Hightereey Crape
Modee Caspeate God
0.4
Malken Antomny Coopastinal Car
LensLLM
0.2
Rectified
4.4. Ablation Study
SubTuning
FullTuning
We perform ablation studies to assess the sensitivity of our method to the stopping criteria criteria- sppciifcalll...l..l....... the regression threshold (7) and the stop threshold (r). The following tables summarize the impact of varying these parameters on the Pearson correlation across three datasets:
0.0
FLOPs Computational Cost
Figure Pareto-Optimality curve between the selection perfor- mance and the computational costs (in units of 102) (Smaller FLOPs means lower computational cost)
Table 5.
Impact of y and r on PearCorr on FLAN
>\r
2
3
4
5
Table Comparison of computational costs (in units of 102!) benchmarks, where LENSLLM achieves superior efficiency with up 88.5% reduction compared baseline methods.
78.41
78.40
78.42
78.43
78.31
4
78.32
78.39
78.36
78.40
78.40
78.39
78.41
78.40
78.39
78.34
Method
Coes
FLANN
Wikitext
Gigaword
FullTuning
6thN
3.35
5.13
4.28
Table 6. Impact of y and 7 on PearCorr on Gigaword
SubTuning
EEM
0.92
1.81
1.11
Cruttaning
7\
2
3
4
5
85,74
85.74
85,80
85.71
85.66
LENSLLM
M.2
0.48
0.59
0.97
4
85.62
85.71
85.75
85.79
85.66
Csartinng
85.64
85.69
85.66
85.76
85.72
We conduct comprehensive efficiency analysis comparing our model's computational requirements with existing meth- ods. Following Kaplan et al. (2020), we estimate the com- putational cost Coost in floating point operations (FLOPs) as Ccvst ~ D, where N denotes the number of model parameters and represents the dataset size. For fair
The results across FL.AN, Gigaword, and Wikitext datasets demonstrate the robustness of our method with respect to the hyperparameters and т, which govern the stopping criteria. For FLAN, the Pearson correlation remains highly stable, with minor fluctuations between 78.31 and 78.43.
8

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
efficiency through meta-learning strategies. Meanwhile, re- cent advances in understanding model generalization (Wang et al., 2024a;b) emphasize the importance of capturing long- tail patterns and dynamic, non-llD conditions. While these methods show promise in specific contexts, they often as- sume static model behaviors and fail to account for the dynamic scaling interactions.
Table Impact of y and
PearCorr on Wikitext
y\T
1
2
3
4
5
3
82.47
82.58
82,47
82,48
82.44
4
82.49
82.51
82.48
82,49
82.54
5
82.61
82.50
82,46
82.57
82.53
Similarly, Gigaword shows consistent performance with correlations ranging from 85.66 to 85.79. Wikitext exhibits slightly more variability, but the range (82.42 to 82.61) still reflects strong stability. These minor fluctuations across all datasets suggest that the performance of our approach is largely insensitive moderate changes in the stopping parameters, underscoring its reliability and robustness in practical settings.
6. Conclusion
This work presents a significant contribution to LLM se- lection by establishing a first- iist-principled framework that bridges the gap between theoretical foundations and em- pirical observations. Our key contributions are threefold: First, we propose a first-principled PAC-Bayesian General- ization Bound that reveals the dynamics of transition phases in fine-tuning, providing a theoretical lens for examining LLM selection and its generalization across tasks. Second, building on this foundation, we introduce LENSLLM, which integrates NTK with scaling laws to better identify the transi- tion mechanism between phases. Third, our comprehensive empirical evaluation demonstrates the exceptional perfor- mance of our approach, achieving up to 91.1% relative ac- curacy and 85.8% Pearson correlation across benchmarks substantially outperforming existing methods. Moreover, our model achieves superior computational efficiency, which reduces computational costs by up to 88.5% compared to FullTuning. Our work establishes a new baseline for LLM selection by achieving an optimal balance between accuracy and efficiency. There might be several promising directions for future research: extending the analysis to multi-task sce- narios, exploring implications for architectural design, and investigating applications to emerging model architectures, e.g. MoE models.
5. Related Work
Early model selection relied on feature similarity meth- ods (Vu al., 2020; Dwivedi & Roig, 2019) predict transfer performance by comparing source and target tasks. However, these approaches couldn't 't capture the complex fine-tuning dynamics revealed by our NTK-based analy- sis, particularly the pre-power and power phases. Recent work has introduced training-free frre transferability metrics, such as LogME (You et al., 2021), which attempt to predict model performance without additional fine-tuning. While these approaches reduce computational overhead, they fail to account for the dynamic nature of fine-tuning that our theoretical framework explicitly models.
The development of scaling laws has been crucial in un- derstanding LLM performance during pre-training. Kaplan et al. (2020) established foundational power-law relation- ships between model size, dataset size, and performance. Extensions of these scaling laws (Ghorbani et al., 2021) demonstrated their predictive power across various scales. Recent work by Isik et al. (2024) has further advanced our understanding by developing scaling laws specifically for downstream task performance, showing how different tasks exhibit distinct scaling behaviors. However, these re- lationships become insufficient during fine-tuning due to the emergence of distinet phases with different governing dynamics. Recent attempts to adapt scaling laws to fine- tuning, such as Rectified Scaling Law (Haowei et al., 2024), have shown promise by incorporating dataset properties and task-specific factors. Zhang et al. (2024) provides a comprehensive analysis of how data quantity, model size, and fine-tuning methods interact during LLM adaptation, revealing complex trade-offs that traditional scaling laws fail to capture. However, these approaches fall short in low- resource scenarios where the pre-power phase dominates.
A Acknowledgements
We thank the anonymous reviewers for their constructive comments. This work is supported by the National Sci- ence Foundation under Award No. IIS-2339989 and No. 2406439, DARPA under contract No. HR001 12490370 and No. HR001 124S0013, U.S. Department of Homeland Security under Grant Award No. 17STCIN00001 -00-000 Amazon- Amazzn-Virginia Tech Initiative for Efficient and Robust Machine Learning, Amazon AWS, Google, Cisco, 4-VA, Commonwealth Cyber Initiative, National Surface Trans- portation Safety Center for Excellence, and Virginia Tech. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government.
Other recent methods, including learning-to-rn -o-rank and measfer-learning-bare based approaches (Ji et al., 2024; Hu & Zhang, 2023), have focused on improving model selection
9

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Impact Statement
DeepSeek-AI. Deepseek IIm: Scaling open-source
language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. URL https:/ /github.
This paper advances machine learning techniques with po- tential applications across various domains. We acknowl- edge several important societal implications of this work. First, our methods could improve efficiency and accuracy in automated secisionnmmaing systems, potentially benefiting fields like healthcare and environmental monitoring. How- ever, we also recognize potential risks, including algorithmic bias if the models are trained on non-representative data and the environmental impact of computational resources re- quired for training. To mitigate these concerns, we provide guidelines for responsible implementation and discuss the importance of representative training data. We encourage future work to further investigate both positive and negative societal impacts, particularly regarding fairness, account- ability, and environmental sustainability.
com/ Seeesseek-aiDeeepeeeee
Dong, Y., Cordonnier, J.-B., and Loukas, A. Attention is not all you need, pure attention loses rank doubly exponen- tially with depth. 2021. URL htttt:////axxii abs 2103. 03404.
Dwivedi, K. and Roig. G. Representation similarity anal- ysis for efficient task taxonomy & transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12387-12396, 2019,
Ghorbani, B., Firat, O., Freitag, M., Bapna, A., Krikun, M., Garcia, X., Chelba, C., and Cherry, C. Scaling laws for neural machine translation, 2021. URL https:// arxiv. wwwiv..or///.. wwwoorrgggaoooo abs 22002.110.. 21109007700....
07740.
References
Haowei, L., Baizhou, H., Haotian, Y., Qinyu, C., Zihao, W., Sujian, L.,, Jianzhu, M., Xiaojun, W., James, 7.,, and Yitao, L. Selecting large language model fine-tune via rectified scaling law. ICML, 2024.
Allen-Zhu,
Zhu, Z., Li. Y., and Song, Z. A convergence the- ory for deep learning via over-parameterizatio. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceed- ings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learn- ing Research, pp. 242-252. PMLR, 09-15 15 Jun 2019. BtRhttps:: http///pooeednnn....e.. mlr press/v97/ allen zhu1 9a. html.
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., Hallacy, C., Mann, B., Radford, A., Ramesh, A., Ryder, N., Ziegler, D. M., Schulman, J., Amodei, D., and Mc- Candlish, S. Scaling laws for autoregressive generative modeling, 2020. URL https: http:p:://aaxx..or/g///// 2010 14701.
Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Étienne Goffinet, Hesslow, D., Launay, J., Malartic, Q. Mazzotta, D., Noune, B., Pannier, B., and Penedo, G. The falcon series of open language models, 2023. URL https: http:://xxivvvooooee org/ abs/ 2/1//311.110 16867.
Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer, 2021. URL https:// a arxiv. wwxiv..rg/aiie absg http//sss/2222.. 211./22222.22222
Hu, X. and Zhang, X. Optimal prrameter-transfer learning by semiparametrie model averaging. Journal of Machine Learning Research, 24(358): 2023. URL http:
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Explaining neural scaling laws. Proceedings of the Na- tional Academy of Sciences, 121(,7), June 2024. ISSN URL http:
/ //jmmr.org//////eeleeeee...llltt www.o//pppp///////.. 2ttp////22220000000 003000000.0 html.
1091-6490. doi: 10. 10.1073/pnas..31 18188112211 //dx.do1/ org / /1/./111100 10w3//0as...../ 2311878121.
Isik, B., Ponomareva, N., Hazimeh, H.,, Paparas, D., Vassil- vitskii, S., and Koyejo, S. Scaling laws for downstream task performance of large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2024.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0725. 2021.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent ker- nel: convergence and generalization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, pp. 8580-8589, Red Hook, NY, USA, 2018. Curran Asso- ciates Inc
Dauphin, Y. N., Agarwala, A., and Mobahi, H. Neglected hessian component explains mysteries in sharpness reg- ularization. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Information Processing Sys- tems, volume 37, pp. 131920-131945. Curran Associates, Inc., 2024. URL https: : arxiv. org abs/abss2/44 10809.
Ji, Y., Li, Z., Meng, R., and He, D. Reasoningrank: Teaching student models to rank through reasoning- knowl- edge distillation, 2024. URL // //aaxiivv... .org/ abs/2410. 055.222...
10

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Jiao, A., Patel, T. P., Khurana, S., Korol, A.-M., Brunke, L.,, Adajania, V. K., Culha, U., Zhou, S., and Schoellig, A. P.
neural networks. Applied and Computational Harmonic Analysis, 59:85-1 2022.
Swarm-gpt: Combining large language models with safe motion planning for robot choreography design, 2023. URL https: http//axvvvvo////// org/ wbs./31/.... 01 059.
Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., Lewis, M., and Zettlemoyer, L. Multilingual denois- ing pre-training for neural machine translation. Transac- tions of the Association for Computational Linguistics, 8: 726-742, 2020.
Ju, H., Li, D., and Zhang. H. R. Robust fine-tuning of deep neural networks with hessian-based generaliza- tion guarantees. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 Proceedings of Machine Learning Research, pp.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
PMLR, 17-23 Jul 2022. URIhhtps:: /ttpas://prrooeee ptt.//eooeeeeeeeoooooooooooeee.c ngs ml ......lr..rrr... press/ v1 / httpj/u22aa.....
Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao. J. Large language models: A survey, 2024. URL, https: https///wxvvo.rrreeeeeee 06196.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language mod- els, 2020. URL https: arxiv. org abs 2001. 08361.
Mistral AL. Mistral Al official website, 2024. URL https: / //mistral.at ai.
Schaeffer, R., Miranda, B., and Koyejo, Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems, 36:55565- 55581, 2023.
Kaplun, G., Gurevich, A., Swisa, T., David, M., Shaley- Shwartz, S., and Malach, E. Less more: Selective layer finetuning with subtuning, 2023. URL https: //arxiv. org/ /ttp://32.0000
See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generator networks, 2017. URL https: / arxiv..xiivv//..///aaaa/aaaaaava org abs / 04368.
Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., and Hajishirzi, H. Unifiedqa: Crossing format boundaries with single qa system. arXiv preprint Stt1:200050070000 2020.
Smith, S. L. and Le. Q. V. A bayesian perspective on gen- eralization and stochastic gradient descent, 2018. URL https: https////aaaa...rgaaaa//a//e http:/aaa//111.////.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN
Steele, J. M.
The Cauchy-Schwarz Master Class: An In- troduction to the Art of Mathematical Inequalities. Cam- bridge University Press, Cambridge, 2004. ISBN 978- 0521547789.
Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W. Narang, S., Yogatama, D., Vaswani, A., and Metzler, D. Scale efficiently: Insights from pre-training and fine-tuning transformers, 2022. URL https://acxiv. http:aawwww..eaa/e////.....eeee.
Lee, S., Lee, J., and Park, S. Lipschitz-certtffable training with tight outer bound. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 16891-16902. Curran Associates, Inc., 2020. URL httpps:/pppcceeeingg..r cc ccc/pppee_fllleepppprr//////1/1 hww.:eedddiiiinnnnisssee.....e. c4 045-8885519722235544444699222889 pdf.
Team, N., Costa-jussä, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E.,, Lam, J., Licht, D., Maillard, J., Sun, A., Wang, S., Wenzek, G., Youngblood, A., Akula, B., Barrault, L., Gonzalez, G. M., Hansanti, P. Hoffman, J.. Jarrett, S., Sadagopan, Rowe, D., Spruit, S., Tran, C., Andrews, P., Ayan, N. F,, Bhosale, S., Edunov, S., Fan, A., Gao, C., Goswami, V., Guzmán, F., Koehn, P., Mourachko, A., Ropers, C., Saleem, S., Schwenk, H., and Wang, J. No language left behind: Scaling human-centered machine transla- tion, 2022. URL https: / /arxiv. org/ wtggabs/////ooooooooo.gggt.cc 2207. 04672.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mo- hamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, I.. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehen- sion. arXiv preprint
R.,
2019.
Liu, C., Zhu, L., and Belkin, M. Loss landscapes and opti- mization in over pee-parameterized non- conelinear systems and

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kul- shreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L.,, Du, Y., et al. Lamda: Language models for dialog appli- cations. arXiv preprint arXiv:2201.08233 2022.
models are zero-shot learners. In The Tenth Interna- tional Conference Learning Representations, ICLR
2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022a. URL https:/ /oppnrevieeeeeeeeeeeeeeeeeeeeeee net forum? i id=gEzrGCozdqR.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint 0002:2307.009288, 2023.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abili- ties of large language models, 2022b. URL https: / / /aarxiv... www./oog/////o/oo/.//...oo..../o ht/abbb//206.. 07682.
Trauger, J. and Tewari, A. Sequence length indepen- dent norm-based generalization bounds for transformers. CoRR, abs/2310. 10.1100..00... 30888, 2023. doi: 10.48550/ARXIV. 2310.13088. URL https:::/// arXiv. 2310 0 13088.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thougee prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022c.
48550/
Villalobos, P., Sevilla, J., Besiroglu, T., Heim, L., Ho, A., and Hobbhahn, M. Machine learning model sizes and the parameter gap, 2022. URL https: /etpsv///avvv.o/o org/ abs/2207 02852.
Wu, Q. Bansal, G., Zhang. J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., and Wang. C. Autogen: Enabling next- gen Ilm applications via multi-agent conversation, 2023. URL https: / / //t./aiivv/////////////.////aava www.riii.......eee.ooo..eeooooee wor.oorg//aaa.. /ttaa/bb//0.... 2308. 08155.
Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarclla-Micke, A., Maji, S., and Iyyer, M. Ex- ploring and predicting transferability across NLP tasks. In Webber. B., Cohn, T., He, Y., and Liu, (eds.), Pro- ceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 7882- 7926, Online, November 2020. Association for Computa- tional Linguistics. doi: 635. URL ht https:://// //cccclaaa //aaaaaaaaanoooyyo/ emn 1amllppmaae 1ampmainn..... in.iin.. 635/.
Yang, Y., Theisen, R., Hodgkinson, L.,, Gonzalez, J. E., Ramchandran, K., Martin, C. H., and Mahoney, M. W. Evaluating natural language processing models with gen- eralization metries that do not need access any training testing data, 2023. URL https: http::::/aaxxxv.rrt a abs / 2202. 02842.
You, K., Liu, Y., Wang, J., and Long, M. Logme: Practical assessment of pre-trained models for transfer learning. In ICML, 2021.
Wang, H., Jing. B., Ding, K., Zhu, Y., Cheng, W., Zhang, S., Fan, Y., Zhang, I.,, and Zhou, D. Mastering long-tail complexity on graphs: Characterization, learning, and generalization, 2024a. URL abs/2305. 09938.
Zhang, B., Liu, Z., Cherry, C., and Firat, O. When seal- ing meets 11m finetuning: The effect of data, model and finetuning method. In The Twelfth International Conference Learning Representations, 2024. URL https: https://arxxv... http://aaxwv...oaaa/////aoa httpg//bbs/222..11133
Wang, H., Mao, Y., Yan, Y., Yang. Y., Sun, J., Choi, K., Veeramani, B., Hu, A., Bowen, E., Cody, T., and Zhou, D. Evolunet: Advancing dynamic non-iid transfer learning on graphs, 2024b. URL / /taaxxiivoorrr org/ /bs/ 2305. 00664.
Zhang, J., Zhao, Y., Salch, M., and Liu, P. J. Pegasus: pre-training with extracted gap- sentenenesees for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
Wang, J. and Duan, 7. Agent ai with langgraph: A modular framework for enhancing machine translation using large language models, 2024. URL https: / foss/212..0 03801.
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang. C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.-Y. and Wen, J.-R. A survey of large language models, 2024. URL https://aaxii: //axxiv.. 18223.
Wei, C., Lee, D., Liu, Q., and Ma. T. Regularization matters: generalization and optimization of neural nets v.s. their induced kernel. Curran Associates Inc., Red
Hook, NY, USA, 2019.
Wei. J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language
12

---

LENSLLM; Unveiling Fine-Tuning Dynamics for LLM Selection
A. Proof of PAC-Bayesian Generalization Bound
This section presents our proof of Theorem We start with problem setup, followed by assumptions. Then we prove the theorem by following two important lemmas.
A.1. Problem Setup
Consider predicting a target task given a training dataset of size n. Denote the feature vectors and labels as æ; and ys, for i l....., n, which æ; d-dimensional and ys is a class label between k. Assume that the training examples are drawn independently from an unknown distribution D.
We define two probability distributions:
(1)A prior distribution P, which represents noisy perturbations around pretrained weight matrices W from the source domain.
(2)A posterior distribution C, which represents noisy perturbations around fine-tuned weights W the target domain.
From a PAC-Bayesian perspective, the generalization performance is determined by how these noisy perturbations affect predictions across domains.
Consider an 1-layer transformer that has been pretrained, with weight matrices W for i = 1,...,,. weights for the target task. Let fw denote an (-layer transformer initialized with the pretrained weights (
I. We then fine-tune these
For each layer i, the transformer contains the following parameters:
(1) Three matrices for attention: Query projection matrix: W Key projection matrix: wkK and Value projection matrix: W..
(2) Two matrices for the feed-forward network: First layer weights: W and Second layer weights: W?
(3) Layer normalization parameters: Scale parameter: "ss and Shift parameter: 3r.
A.2. Assumptions
Our analysis builds upon several carefully constructed assumptions that connect classical DNN theory with transformer architectures:
Assumption 1 (Smoothness). The activation functions o; on layer and loss function L are twice difce-diffeenttaalle with Lintchis first and second derivatives (Allen-Zhr) -hu et al., 2019):
(I) (x) Vo:(u)ll < LøIl - yll. where Ls is the Lipschitz constant that bounds how fast the gradient of Φ; can change;
(2) [VVo, ( L x yll. where Lgr the Lipschitz constant for the second derivative of oi. Assumption (Boundedness). Following Wei al. (2019), the loss function and input features satisfy uniform bounds: L L C and
|=|||| 2 < B.
Assumption (Transformer Stability). Based on Liu al. (2022), transformer components satisfy:
(1) Attention mechanisms are L Lipsshhti continuous (Lee al., 2020);
(2) Attention scores are bounded: ||softmax (QKT) oox S M (Dong et al., 2021);
(3) Residual connections maintain gradient flow: [x) 28> 0;
(4) Layer normalization preserves scale: ||LN (x) = ||0||2.
A.3. Proofs for PAC-Bayesian Generalization Bound
Before we start to prove Theorem 2, we will need the KL divergence between the prior and the posterior Q in the PAC-Bayesian analysis. This is stated in the following result:
Proposition (KL Divergence for Transformer). Suppose the noise perturbation at layer is drawn from a Gaussian distribution with mean zero and covariance Σ,, for every =,...., Then:
13

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
1. The KL divergence between P and is:
(L(QIP) = 2 1=1
Σ
(., w (W, w
(LI)
2. For isotropic noise distribution every layer (i.e., E; 0,
W W (+)
(T
(12)
20
1=1
Proof. The proof follows from standard results multivariate normal distributions with additional attention to transformer components.
Let Z: be the weight matrix of layer in the posterior distribution. By definition of KI. divergence:
KL(QIP) Ez~e log 2(
P(Z)
Ezzzzloog flog Q(2) log P(o))()) (Z)
Σ
1
1 vecl Zi W.) Σ, vee(Z W.) + veel Z W 'e'vee((2 vec( - w
Ezne
2
2
wheer Z. includes both feed-forward and attention parameters on layer i. Computing the expectation over Zi and using matrix trace properties:
KL(2|P) --2 -EEZ
Σ Tr |uec Z
W,)oec(2, Wi) [ Tr veec Z; W (ecc)ec Z TT)")
Given that the expectation Zg is W; and covariance Ei, after cancellation:
KI(Q 1 Σ vec(WW W. -W(ol) Σ, Seceec W
For isotropic case where E, = IdId, this simplifies to:
W. ('(
KL
Σ
20
1=1
Theorem 2 (PAC-Bayesian Generalization Bound). For a fine-luned transformer model f satisfying Assumptions 1-3, with probability at least 0.99 and any fixed 0:
(c+ 1)(())++ (1+E)OEL VT.
[(fo) s (c+
vn + O0n
(4)
where h; 2 max oi H (L.()))))
14

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Proof. First, separate the gap between L(fe) and (L(()) into three parts:
1 - (f(( + 1 - (it
(lfa) ((f)
(13)
By Taylor's expansion, can bound Equation 13 with respect to the empirical loss and the expected loss:
1 L(fa) 3 [(fa) - -B(a)))
V H (((f((a).))))
+ Σ C, IV
-
+ 1
v H ((ff((()))
E.I.H.
j=1
which simplifies to:
((fa) 1 (((f)) (B, -B
1
VH ((foo()))) nB
V H. [e(f (z)). v,)1)
(
Σ C. I. (Le(fa) - fLol(el)
Next, we combine the upper bound on the noise stability of fa with respect the empirical loss and the expected loss:
1
Hi (((Fa
(AWND
Σ HV (((fa())
n2
V H: [((fo(;)) ((aa)) (VV [e(ff
3
n j=l
+ (
v (E (a (((f(()), v))))
Recall that u, is flattened vector of the matrix W. - w(()) By the assumptions 1-3 and the KL divergence proposition,
1
(((s) < C(KL(Q)P)+ + 1
[(fa)
B
2 ()
C CbE (v,, E, 1,'1,) + log /
2 - B)m
Combining all above equations with probability at least 28, we get the following Inequation 14:
1 1 i((fa)
C2 CoyTog(Can/0//n
3 -1 (1-1) V, (B( (e(f(())
I(fa) B
5
V H || +
3
(14)
2 E/= (v, E,'o, 0,) + log }
(3
( C1 Σ
IV
2 B)m
15

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Recall the truncated Hessian H [e(s (a)), y)] is equal to U, max O)U," where U,D,U! the eigen-decompositionn
of H, (e(fa( (2:), y)]- For any (xc, y) D, we have:
V H, (((F(z) S H. [((f(()))))
Thus, after taking ((,yy) both sides, have the following inequation:
(
C BBa H (((((x)))))))) ),,o))
V. (B. (())(())))))
C
V ///21//u./1?
(B H (x) will
willlo?
(15)
Co. ((.
H [((fa()), y)]l
is
C74
4
g
Next, based on the Inequation 15, the upper bound of I(fa) (fxx) could be extended to the following:
1
L(fa)
(ff))
3
CH, 82n
log( (Cnn/8//n
1
v Hi + (1+
log 1
+
C2
3 C1
V
B
23(1
m
8
1=1
Then we set = (1 - 3, and get this:
CHi
Uff=) S (1 + (i(((()))
f2n
log( Tog(Can/0)/n
V Hi + (1+ C Σ
C
Hi 3/2 +
log
B
23
3)n
where the last part is equal to On 2). Hence, conclude that:
CHi V B2n
L(fa) ((+
Σ
+ O(n +).
(16)
Thus we have shown the Theorem 2 holds.
This result shows that transformer architectures maintain PAC-Bayesian generalization guarantees with additional terms accounting for attention mechanisms, layer normalization, and residual connections.
B. Proof of Scaling Behavior in PAC-Bayesian Generalization Bound
In this appendix, we provide the detailed proofs for corollary 1.
16

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
B.1. Property 1: Cauchy-Schwarz hg
Lemma 1. E Vh S v Ei=1
Proof. The proof follows these key steps:
1) First, recall the Cauchy-Schwer inequality: For vectors u, E R", (u, v)) S |||||/|]|-
2) Let's define our vectors:
u (Vhi, Vh2,.... Vhn)
v = (I,1,...,, 1) (n-dimensional vector of ones)
3) Calculate the left side (u, w}|;
(u, v) = Σ
v h4 1)
==1
n
Σ h4
4) Calculate ||/]|:
h1. 2
1=1
Σ 1=1
h:
5) Calculate ||m||]
|||||
Σ
12
1=1
Vn
6) Apply Cauchy-Schward
(14, 1) S ||
Vhi
hi vn
i=1
Σ Vh
Σ h4
Therefore, we have proven that E Vh, S vn Σm h4.
Note: This proof assumes all h; are nen-negative real numbers, which is aligned with the property of h4 in our bound.
B.2, Property 2: Upper Bound of h;
We first prove that the sum the trace of the Hessian matrix each layer tr (H) in a transformer model, when summed across all layers, proportional -8 where the size of the fine-tuning dataset and B is constant.
17

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
B.2.1. SUM OF TRACE OF H;
Define the Objective Function and Hessian
Let L(0) be the loss function the model parameterized by 8.
The Hessian matrix for this loss function is defined as:
H H (8)
(17)
For each layer I, let H, be the Hessian of the loss function with respect the parameters in that layer.
The sum of the trace the Hessian matrix across all layers is:
Σ
tr ( - HH(
(18)
2. Scaling Behavior of the Hessian with Respect to Dataset Size
Lemma 2. L(0) is the empirical loss over dataset of size n, the trace of the Hessian matrix H scales as tr (r())
n- V--Var(VL(O))
Proof.
Consider the empirical loss function:
1 n
(( = Σ ((0;,;)
(19)
where ((;; x4) is the loss associated with sample Ti.
The Hessian of this loss function can be expressed as:
H
(20)
n
where H, = x0) is the Hessian of the individual sample loss.
By taking the trace, have:
1
ur( H)
tH.)))
(21)
n
Using the Central Limit Theorem (CLT) and assuming that H; are ii.d., the variance of the gradient VE(0; xx) becomes:
1 Var Var VE(())
Var Var(V)(())
(22)
From Dauphin et al. (2024), we have the following relation between the trace and Var(VL(()):
Hr( = Var(VV(()) + tr (V.L- Viiz)
(23)
if we assume that the model "fits" the data, w.r. VoL(0*) = 0, then we could get the following result:
tr H = Var(v
(24)
That is:
tr H = n Var(VLL())
(25)
18

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
3. The behavior of Variance During Fine-Tuning
Lemma 3. During fine-tuning, the variance the
the gradient scales as Var(VL((0) n2" for some constant a.
Proof.
Let the variance of the gradient during fine-tuning be d2
Empirical observations and theoretical results from the literature suggest that as the size of the dataset increases, the variance of the gradient decreases. By analyzing the dynamics of SGD through a Bayesian lens, this behavior is often modeled as Smith & Le (2018):
a² (n) & n
(26)
The parameter characterizes the behavior of the variance reduction. It is typically derived based on the structure of the model and the type of regularization applied.
Combining Lemmas and 2
From Lemma 1:
tr(H) = n -n-o n=2(2(n))
(27)
Substituting the result from Lemma 2:
tr (r(H) x xn nn nn
(28)
The sum of the trace the Hessian matrix across all layers proportional to n where 8 = a + tr( (r( Cin the conclusion of this statement.
Let's give
B.2.2. UPPER BOUND OF h;
Based proof in Appendix A, we have the following formula for L-layer transformer:
L(fa) S (c+ (1 (1+ (1+, Vh:
+
(29)
Vn
where h; is any value greater than max of H (e(fa(x). v) ug for 1,...,I. L. To make the proof more straightforward and clear, we would start our proof 2-layer transformer follows:
L(fa) & (c+ 1)i(f)))))+++ + (1 ++OV Vh
+
(30)
Vn
where h; is any value greater than max of H (e(fa(x). y) ur for all 1,...,2. 2.
We then prove that the sum of hy has an upper bound C2n na, given the distance-based generalization, where C2 is independent of n.
Define the Objective Function and Weight Matrix
Let (+) be the weight matrices of pre-trained model and W, be the dimension of layer i. The dimension W, is d, by d,-1, where is the dimension of input 24.
The distance-based regularization is defined as for every layer:
W: W 1| a
0z, Vi 1,2
(31)
Therefore, hy can be forward defined as:
ha W: ( max (( (x.0)ex
(32)
19

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Upper Bound for I[W
Lemma 4, There exists an upper bound for W; w [8: which is unrelated the training data size:
W; W 11 S B
(33)
Proof. Trauger& Tewari (2023) has proved that for any E
* Σ
I1(W
W) W
((W
Wx,) 8 S kc
(34)
j=1
From which, we could derive:
IIW W)xlll - ke
(35)
For any 2 + 0 we have:
he 2
( W)13
(36)
ax )
in our setting.
Thus, considering the relation between Frobenius Norm and Spectral Norm, we have:
I( w( K
(37)
min (a;aa)
Thee, we have:
o²
W; w S min(di d
B
(38)
Since none of d;, and æ; relies on training data size n, C2 is independent of n.
When æ; = 0, we have:
(, =
(39)
This because there no training data for fine-tuning, the weights would not change, and 0 is independent of n.
Thus,
[[W, W < B
(40)
works for any Zz.
Since IW ( || has upper bound of B, we can derive a upper bound h; as max Tr HT(Hf(((fa (x), v))] Using the result of statement we can continually derive the upper bound as:
h4 S Can 82
(41)
B.3. Fine-tuning Scaling Law
Since H; is always larger than O, then extend the above bound the following by using the relation of
Ea Vhi S V2
(1+ h
U(fa) s (E+ 1)((f))
+
(42)
20

---

LENSLLM; Unveiling Fine-Tuning Dynamics for LLM Selection
From the above statements, finally reach our conclusion as follows:
s(++ (1+OVOV 2 2 hi
+
Vn
(1+ (L+ 202m
(43)
s 1)L(fa)
+
Vn
S ( 1)+()))(m))))))))))))))))) Cyn -mm O(n
This final result can also extend to l-layer transformer as follows:
(c+))(()))) (
Vn +
( 1)ū(fo) + (1+c)VC VIC2n 8
(44)
+
Vn
s {(e+1)L(Jo) Can 92 O(
Thus, we could finally derive the upper bound of the test loss scaling behavior as follows:
L(f=) S (c+ 1)f( Can O(n
(45)
C. Additional Experimental Results
All fine-tuming was performed by using PyTorch and the Hugging Face Transformers library.
C.1. Sensitivity to Hyperparanters
To further evaluate the robustness of our fine-tuning process, conducted series of experiments analyzing the sensitivity to key hyperparameters, including learning rates, batch sizes, and average input sequence length. In particular, we examined how variations in learning rate and batch size influence the Pearson correlation across three benchmark datasets. The results of these experiments are summarized below.
Table Impact of Learning Rate and Batch Sizes Pearson Correlation FL.N
Batch size
Learning rate
3e -5
le
3e -4
1c
64
78.36
78.41
78.40
78.39
128
78.32
78.34
78.43
78.36
256
78.37
78.36
78.36
78.34
Table 9. Impact of Learning Rate and Batch Sizes on Pearson Correlation on Gigaword
Batch size
Learning rate
35-5
le
3e
1e
64
85.74
85.73
85.74
85.75
128
85,74
85.79
85.77
85.76
256
85,69
85.72
85.71
85.69
The results demonstrate that method exhibits strong robustness with respect to variations in learning rates and batch sizes. For the FLAN dataset, Pearson correlation remains highly stable, fluctuating only slightly within the range of 78.32 to 78.43. Similarly, the Gigaword dataset shows consistent correlation values between 85,69 and 85.79 across all configurations. Although the Wikitext dataset exhibits marginally more variability, the correlations still remain tightly clustered between 82.50 and 82.61. These findings indicate that the model's 's performance is largely insensitive to changes in these hyperparameters, reinforcing the reliability our fine-tuning process.
21

---

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Table 10. Impact of Learning Rate and Batch Sizes Pearson Correlation Wikitext
Batch size
Learning rate
3e
5
le 4
3e
1c 3
64
82.60
82.61
82.60
82.60
128
82.54
82.53
82.55
82.55
256
82.51
82.51
82.51
82.50
C.2. Sensitivity Input Length
To assess the impact of average input sequence length, we adjusted the input distribution by removing either the shortest or longest sequences, resulting in average lengths of 18 and 22 tokens, respectively, compared to the original average of 20. As shown in Table 11, both shorter and longer averages lead slight decreases in Pearson correlation and relative accuracy. Specifically, the Pearson correlation drops from 78.14 14 (at average length 20) to 77.39 and 76.89 for lengths 18 and 22, respectively, while relative accuracy similarly declines. Although performance is affected by these changes, the overall variation remains small, suggesting that the model is relatively robust to moderate fluctuations in input sequence length.
Table 11. Impact Average Input Sequence Length on FLAN
Metric \ Avg. Input Seq. Length
18
20
22
Pearson Correlation (PearCorr)
77.39
78.14
76.89
Relative Accuracy (RelAcc)
87.86
88.88
87.91
22


Reviews:
[{"review_id": "fvR6YcQk44", "reviewer": "Reviewer_2LaB", "summary": "LensLLM introduces a novel theoretical framework that addresses the fundamental challenge of efficient Large Language Model selection through the lens of fine-tuning dynamics. The paper develops a rigorous Hessian-based PAC-Bayes generalization bound that characterizes two distinct phases in LLM fine-tuning: a \"pre-power phase\" in low-data regimes where performance improves slowly due to high Hessian values and parameter sensitivity, and a \"power phase\" where improvements follow predictable power-law scaling with enhanced stability. The authors implement this theoretical insight through a Neural Tangent Kernel (NTK)-based Rectified Scaling Model that accurately predicts model performance across diverse tasks while maintaining computational efficiency. Empirical evaluation across three benchmarks (FLAN, Wikitext, and Gigaword) demonstrates the framework's effectiveness, achieving up to 91.1% relative accuracy and 85.8% Pearson correlation while reducing computational costs by up to 88.5% compared to full fine-tuning approaches. The work establishes a new foundation for understanding LLM generalization during fine-tuning and provides practitioners with a principled approach to model selection under computational constraints.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: How sensitive is the method to hyperparameter choices, particularly in the algorithm's stopping criteria?\n\nA1: Thank you for your question. \n\na. We performed ablation studies to assess the sensitivity of our method to the stopping criteria—specifically, the regression threshold ($\\gamma$) and the stop threshold ($\\tau$). The following tables summarize the impact of varying these parameters on the Pearson correlation across three datasets:\n\nTable1: Impact of $\\gamma$ and $\\tau$ on PearCorr on FLAN\n|$\\gamma$\\PearCorr\\ $\\tau$|1|2|3|4|5|\n|-|-|-|-|-|-|\n|3|78.41|78.40|78.42|78.43|78.31|\n|4|78.32|78.39|78.36|78.40|78.40|\n|5|78.39|78.41|78.40|78.39|78.34|\n\nTable2: Impact of $\\gamma$ and $\\tau$ on PearCorr on Gigaword\n|$\\gamma$\\PearCorr\\ $\\tau$|1|2|3|4|5|\n|-|-|-|-|-|-|\n|3|85.74|85.74|85.80|85.71|85.66|\n|4|85.62|85.71|85.75|85.79|85.66|\n|5|85.64|85.69|85.66|85.76|85.72|\n\nTable3: Impact of $\\gamma$ and $\\tau$ on PearCorr on Wikitext\n|$\\gamma$\\PearCorr\\ $\\tau$|1|2|3|4|5|\n|-|-|-|-|-|-|\n|3|82.47|82.58|82.47|82.48|82.44|\n|4|82.49|82.51|82.48|82.49|82.54|\n|5|82.61|82.50|82.46|82.57|82.53|\n\nObservations:\n1. FLAN: Pearson correlations remain stable (approximately 78.31 to 78.43) across different values of $\\gamma$ and $\\tau$.\n2. Gigaword: The correlation values are consistent, ranging from about 85.66 to 85.79.\n3. Wikitext: A slight variation is observed, with correlations fluctuating between roughly 82.42 and 82.61.\n\nOverall, the minor fluctuations in Pearson correlation across different $\\gamma$ and $\\tau$ values indicate that our method is robust concerning the stopping criteria. \n\nb. We also conducted some additional experiments to test the sensitivity to hyperparamters in fine-tuning process - specifically, learning rates, batch sizes and average input sequence length. \n\nThe impacts of learning rates and batch sizes on the Pearson correlation across three datasets are summarized below:\n\nTable4: Impact of Learning Rate and Batch Sizes on PearCorr on FLAN\n|Batch size\\Metric\\Learning rate|3e−5|1e−4|3e−4|1e−3|\n|-|-|-|-|-|\n|64|78.36|78.41|78.40|78.39|\n|128|78.32|78.34|78.43|78.36|\n|256|78.37|78.36|78.36|78.34|\n\nTable5: Impact of Learning Rate and Batch Sizes on PearCorr on Gigaword\n|Batch size\\Metric\\Learning rate|3e−5|1e−4|3e−4|1e−3| \n|-|-|-|-|-|\n|64|85.74|85.73|85.74|85.75|\n|128|85.74|85.79|85.77|85.76|\n|256|85.69|85.72|85.71|85.69|\n\nTable6: Impact of Learning Rate and Batch Sizes on PearCorr on Wikitext\n|Batch size\\Metric\\Learning rate|3e−5|1e−4|3e−4|1e−3|\n|-|-|-|-|-|\n|64|82.60|82.61|82.60|82.60|\n|128|82.54|82.53|82.55|82.55|\n|256|82.51|82.51|82.51|82.50|\n\nObservations:\n1. FLAN: Pearson correlations remain stable (approximately 78.32 to 78.43).\n2. Gigaword: The correlation values are consistent, ranging from about 85.69 to 85.79.\n3. Wikitext: A slight variation is observed, with correlations fluctuating between roughly 82.50 and 82.61.\n\nOverall, our method is robust concerning the learning rates and batch sizes. \n\nDue to time constraints, we evaluated only the average input sequence length on FLAN while keeping the optimal learning rate (3e-4) and batch size (128) as determined earlier. In FLAN, the overall average input sequence length is 20. To test the effect of altering this average, we removed either the shortest or longest sequences to adjust the average to 18 and 22, respectively.\n\nTable7: Impact of Average Input Sequence Length on FLAN\n|Metric/Average Input Sequence Length|18|20|22| \n|-|-|-|-|\n|PearCorr|77.39|78.14|76.89|\n|RelAcc|87.86|88.88|87.91|\n\nWe observe that the deviations—either shorter (18) or longer (22)—lead to lower Pearson correlation and relative accuracy, the performance gap is not large, which suggests that while there is some sensitivity to sequence length, the model remains reasonably robust.\n\nPlease let us know if there are any comments or insights, we'd like to explore further!", "claims_and_evidence": "The claims are well-supported.", "methods_and_evaluation_criteria": "Evaluation criteria makes sense.", "theoretical_claims": "I checked the correctness of Theorem 2, Corollary 1, and Proposition 1.", "experimental_designs_or_analyses": "The paper presents interesting findings, though there are several areas where the experimental design could be strengthened in future work. The data sampling strategy, which created smaller datasets by \"randomly sampling examples ranging from 200 to 1,638,400,\" would benefit from additional details about whether multiple random samples were used, if sampling preserved the original data distribution, and how many repeated trials were conducted. Similarly, including confidence intervals or statistical significance tests would provide stronger support for the performance differences observed between the method and the baselines.\n\nThe paper also could benefit from expanded analysis of hyperparameter sensitivity. While Algorithm 1 references a regression threshold γ and stop threshold τ, it would be useful to example results might vary with different parameter values. An ablation study would help demonstrate the robustness of this promising approach across different conditions. Despite these limitations, the core methodology seems sound, and addressing these experimental design considerations would further validate the paper's contributions.", "relation_to_broader_scientific_literature": "The proposed LensLLM framework expands upon existing theoretical foundations in machine learning, presenting significant advancements through three primary contributions. Building upon McAllester's foundational PAC-Bayesian theory and Ju et al.'s generalization bounds, this work extends these theoretical constructs to transformer architectures by developing a Hessian-based generalization bound that accounts for transformer-specific elements such as attention mechanisms and layer normalization. This theoretical contribution is notable because it addresses the challenge of analyzing complex transformer architectures that previous theoretical frameworks were not designed to accommodate. In addition, the identification of distinct \"pre-power\" and \"power\" phases in fine-tuning enhances our understanding of scaling laws beyond the established work of Kaplan et al. and Hernandez et al., offering a theoretical explanation for empirically observed behaviors in low-data regimes that previous methods could not adequately characterize.\n\nOn the practical front, the paper's NTK-based Rectified Scaling Model demonstrates the application of NTK theory to finite-width transformers for performance prediction. This model represents a significant improvement over previous approaches such as Lin et al.'s Rectified Scaling Law or You et al.'s LogME, which primarily focused on empirical scaling or feature similarity without capturing the dynamic nature of fine-tuning. The progressive sampling strategy employed achieves substantial computational efficiency improvements compared to existing methods like Kaplun et al.'s SubTuning, effectively addressing the increasingly important challenge of resource-efficient LLM deployment. This combination of theoretical depth and practical efficiency addresses the growing need for principled model selection.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "n/a", "questions_for_authors": "1. How sensitive is the method to hyperparameter choices, particularly in the algorithm's stopping criteria?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YptZw4adw5", "reviewer": "Reviewer_xMun", "summary": "The paper first derives a Hessian-based PAC-Bayes generalization bound that provides deep insight into the fine-tuning dynamics of large language models. It then introduces LENSLLM—a Rectified Scaling Model based on the Neural Tangent Kernel (NTK)—which demonstrates impressive accuracy in predicting performance across a wide range of tasks while maintaining computational efficiency.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions. The followings are our answers to your concerns. \n\nQ1: Could you please clarify the architecture of your regression model?\n\nA1: As illustrated in Section 3.2, our regression model is constructed based on the NTK matrix as follows:\n$$\nL(D) = \\frac{B}{F(\\Theta, t) + D^\\beta} + E\n$$\nwhere \n\n-$F(\\Theta, t)$ is the adapted NTK-based test loss function on transformer\n\n-$D$ is the number of training data\n\n-$\\beta$ denotes the learning difficulty\n\n-B adjusts the initial test loss\n\n-E denotes the optimal loss of the model given an infinite amount of data. \n\nThey are all model/task-dependent and we estimate ${B, E, \\beta, t}$ for each model by minimizing the loss function:\n$$\n\\min_{B,E,\\beta,t} \\sum_{ i} \\text{LSE}(\\log B - \\log(F(\\Theta, t) + D_i^\\beta), \\log E) - \\log L(D_i))\n$$\nWhere $L(D_i)$ denotes the test loss of fine-tuning on the data size $D_i$, and LSE denotes the log-exp-sum operator. \n\nQ2: Is it feasible to train the regression model on one dataset and test it on another? \n\nA2: In our current approach, we leverage the NTK-based Rectified Scaling Model to identify a transition point in the fine-tuning process, after which we perform regression on the loss trajectory. This regression model is then used to predict performance which assist in model selection, particularly in resource-constrained scenarios.\n\nCross-dataset validation, however, poses a significant challenge. The primary difficulty is that the transition point identified on the training dataset may not align with that of a test dataset, due to differences in data distributions and task characteristics. This discrepancy introduce additional challenges to theoretically understand the scaling behaviors of LLM in fine-tuning. This is beyond the scope of this work, and we would like to leave it as our future work.\n\nQ3: Could you please add Rectified Scaling Law performance in Figure 4.\n\nA3: We would like to note that we have already included the performance of the Rectified Scaling Law in Figure 4 (https://anonymous.4open.science/r/LENSLLM-3B1E/Revised%20plot4.png) and will ensure that in the revised manuscript. \n\nQ4: Could you discuss whether your proposed method remains effective when the test model is not included in the training set?\n\nA4: While our experiments primarily focus on evaluating performance within the same set of candidate models, the NTK foundation capturing dynamic behaviors of LLMs during fine-tuning suggests its generalization ability to unseen models, especially those with similar architectural properties and scaling behaviors. To validate this, we conducted additional experiments on the FLAN dataset using LaMini-GPT-774M, GPT2, and BART-large as test models, with LaMini-GPT-124M serving as the training model.\n\nTable: RMSE between predicted and actual test losses\n|Model|RMSE| \n|-|-| \n|LaMini-GPT-774M|1.23| \n|GPT2|1.55| \n|BART-large|5.31| \n\nThe results indicate that our method remains effective when the test model shares similar architectural properties and scaling behaviors with the training model (e.g., LaMini-GPT-124M vs. LaMini-GPT-774M and LaMini-GPT-124M vs. GPT2), which is further supported by our theoretical foundation. Please let us know if there are any comments or insights, we'd like to explore further!\n\nQ5: Could you provide some discussion or how to extend your method to vision-language model selection?\n\nA5: Our current approach models fine-tuning dynamics using the NTK matrix and scaling laws to capture training dynamics and Hessian properties in language models. Extending this framework to vision-language models presents several challenges:\n1. Cross-Modal NTK Formulation: The NTK must be adapted to capture interactions between tokenized image representations (e.g., VQ-VAE or CLIP features) and textual tokens, reflecting joint feature spaces and inter-modal attention.\n2. Modality-Specific Scaling: Vision and language modalities have different scaling behaviors, requiring recalibration of the scaling laws to account for distinct gradients and effective parameter counts, as well as synergy and competition between modalities(Armen Aghajanyan et al., 2023).\n3. Theoretical Bound Adjustments: Our current PAC-Bayesian and NTK-based bounds are tailored to language models; for vision-language models, these would need to be re-derived or adjusted to include modality-specific properties.\n\nDue to these complexities, extending our method to vision-language model selection is non-trivial and beyond the scope of this work, but it remains a promising direction for future research.\n\nPlease let us know if there are any comments or insights, we'd like to explore further!", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The theoretical part looks correct and solid.", "experimental_designs_or_analyses": "The experimental part and analyses are comprehensive.  The performance comparison part looks well. However, the analysis section requires approval. For example, it should assess the effectiveness of the stop threshold ($\\tau$) as well as the computational cost for various model sizes ($M$).", "relation_to_broader_scientific_literature": "Improving the performance of Rectified Scaling Model and also theoretical contribution for PAC-Bayes Generalization Bound.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Overall, the paper looks solid in both theoretical and experimental results. I only have some minor questions.\n\n1. Could you please clarify the architecture of your regression model? \n\n2. If I understood correctly, your experimental setup involves randomly selecting all datasets for the training set. Is it feasible to train the regression model on one dataset and test it on another?\n\n3. Could you please add Rectified Scaling Law performance in Figure 4.\n\n4. Could you discuss whether your proposed method remains effective when the test model is not included in the training set?\n\n5. Could you provide some discussion or how to extend your method to vison-language model selection?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SMaQ0UeRML", "reviewer": "Reviewer_87st", "summary": "The paper proposes a framework, called **LensLLM**, for predicting and selecting the best large language model (LLM) to fine-tune under computationally constrained scenarios. It introduces a theoretical foundation using a Hessian-based PAC-Bayes generalization bound to illustrate how fine-tuning progresses through a “pre-power” phase (where performance improves slowly under low-data regimes) and a “power” phase (where the model follows a more predictable scaling law as dataset size increases). Building on this, the authors design LensLLM—a rectified scaling approach that integrates Neural Tangent Kernel (NTK) concepts with the dynamics revealed by their theoretical analysis. Their experiments suggest that LensLLM not only achieves higher accuracy than existing model-selection methods (e.g., rectified scaling laws, zero-shot, or heuristic-based metrics) but also significantly reduces computation time by progressively sampling smaller portions of the dataset for predictions. Empirical results on FLAN, Wikitext, and Gigaword benchmarks indicate strong correlation and ranking ability while cutting fine-tuning FLOPs by more than half.\n\n## update after rebuttal\nI have no further questions and provide my final rating based on the overall assessment of the paper. A higher rating was not given due to the paper’s limited contribution in comparison to the rectified scaling law.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful questions. The followings are our answers to your concerns. \n\nQ1: How robust is the identified transition point between pre-power and power phases to hyperparameter choices (e.g., learning rate, batch size, sequence length)? Do small changes in these settings significantly shift the transition point or degrade predictive accuracy?\n\nA1: \na. We would like to point out that the transition point is derived from the fine-tuning results as illustrated in pseudo-code in Section3.2, so its stability is closely linked to the robustness of the fine-tuning process. We conducted additional experiments on FLAN with the following setting to test the robustness of fine-tuning:\n\n-Learning rates in (3e−5, 1e−4, 3e−4, 1e−3)\n\n-Batch sizes in (64, 128, 256)\n\n-Average input sequence lengths in (18, 20, 22).\n\nTable: Variance of fine tuning results on FLAN.\n|Model|Variance|\n|-|-|\n|OPT-6.7B|0.0016|\n|T5-Base|0.0022|\n|Cerebras-1.3B|0.0012| \n|MT5-Large|0.0023| \n|BART-Large|0.0042| \n|GPT-2|0.0038| \n|LaMini-774M|0.0026| \n\nThe small variance ranging from 0.0012 to 0.0042 demonstrates the robustness of fine-tuning process, thereby supporting the stability of the identified transition point.\n\nb. We illustrated the robustness of our method to hyperparameters,including regression threshold, stop threshold, learning rate, batch size and sequence length. Due to space constraints, please refer to the rebuttal of reviwer 2LaB.\n\nQ2: For extremely large models, do you expect Hessian approximations to remain stable in practice? Or is there a risk that computing these approximations or performing partial fine-tuning becomes infeasible?\n\nA2: We would like to point out that our approach does not rely on explicitly computing Hessian approximations for predictive modeling. Instead our scaling law prediction model is based on extracting the Neural Tangent Kernel (NTK) matrix, which effectively captures interactions between the data and model features. This allows us to make predictions about fine-tuning behavior without requiring direct Hessian computation. Thus, the Hessian is primarily used in our work to establish a theoretical Bayesian bound, which helps verify the pre-power and power phases from a theoretical perspective. In this case, we leverage general properties of the Hessian rather than computing it explicitly. \n\nQ3: How would a significantly modified Transformer architecture (e.g., MoE, encoder-decoder) impact lens-based predictions? Would your approach require re-fitting theoretical parameters for such architectures?\n\nA3: We would like to point out that we have evaluated both decoder-only (e.g., OPT, GPT-2, LaMini, Cerebras) and encoder-decoder (e.g., T5, mT5, BART) models without re-fitting theoretical parameters in our experiments, demonstrating the consistency of our approach across these architectures.\n\nHowever, for architectures like MoE—where only a subset of parameters is active per forward pass—the effective parameter count is lower. This requires re-fitting the scaling law parameters in our NTK-based framework, specifically by adjusting the effective network width and recalibrating the scaling constants and exponents to account for the sparsity and computational cost. Due to resource constraints, we have not yet conducted experiments on MoE models, but this remains a promising direction for future work.\n\nQ4: What is the exact set of models used? It appears that the models tested are generally not very large—could you clarify why? Additionally, do any of the fine-tuning results come directly from the rectified scaling law, or were they trained independently? If the latter, what are the fine-tuning details, including software and hardware specifications?\n\nA4: \na. As shown in Table 2, the following is our model set:\n|Model Set|\n|-|\n|OPT-350M, 1.3B, 6.7B|\n|T5-Small, Base|\n|Cerebras-256M, 1.3B|\n|MT5-Base, Large|\n|BART-Base, Large|\n|GPT-2|\n|LaMini-124M, 774M|\n\nb. Due to limited access to high-end GPUs (we only used a single A100-80G), we were unable to extend experiments to larger models.\n\nc. The fine-tuning results are all trained by ourselves, and the followings are the software and hardware details:\n\n1. Software: Fine-tuning was conducted using PyTorch with the Hugging Face Transformers library. We use AdamW optimizer and weight decay as 0.01. \n2. Hardware: Experiments were conducted on a single A100-80G.\n\nWe will clarify the above points in the revised manuscript.\n\nA5. We would like to clarify that replacing WMT19 with Wikitext is due to computational constraints, as it is 20 times larger than FLAN and Gigaword. Additional experiments conducted on a randomly selected 1/20-size subset of WMT19 yielded results consistent with our original results.\n\nTable: Model selection performances of our model and Rectified Scaling Law\n|Metric/Method | LensLLM | Rectified Scaling Law |\n|-|-|-|\n|PearCorr|85.7|79.3|\n|RelAcc|90.2|89.0|\n\nPlease let us know if there are any comments or insights, we'd like to explore further!", "claims_and_evidence": "**Claim:** The authors claim that modeling fine-tuning with a Hessian-based PAC-Bayes approach clarifies how model performance transitions from “pre-power” to “power” phases.\n\n- **Evidence:** They provide theoretical reasoning (an extension of the PAC-Bayes style bound) and highlight how truncated Hessian values decrease as more data is used, driving the phase transition. This part is largely conceptual, building on known results and augmenting them for large-scale transformer architectures.\n\n**Claim:** The paper asserts that LensLLM achieves up to 91.1% accuracy in ranking the best fine-tuned model across multiple tasks.\n\n- **Evidence:** The authors compare their method’s selection performance (via Pearson correlation and relative accuracy) against five baselines. The results are consistent across three datasets and multiple model families, demonstrating a clear improvement. The reported metrics show healthy margins over competing methods.\n\n**Claim:** The authors state that LensLLM reduces computational costs by up to 88.5% relative to fully tuning every model on the entire dataset.\n\n- **Evidence:** They provide FLOP-based calculations for each approach (e.g., full fine-tuning vs. partial fine-tuning vs. LensLLM’s iterative sampling). The step-wise sampling procedure indeed appears to require fewer training passes than a naive “train everything fully” approach. The calculations and comparisons are largely in line with standard estimates of training costs.\n\nOverall, the evidence for these claims seems credible, supported by both theoretical discussion and consistent empirical demonstrations across different tasks and architectures.", "methods_and_evaluation_criteria": "- The proposed method uses an NTK-based scaling law and Hessian insights to predict final fine-tuned performance from partial data. This approach is well-motivated: the paper grounds it in the theoretical transition between small- and large-data regimes, something conventional scaling laws often ignore.\n\n- The evaluation criteria focus on ranking accuracy (Pearson correlation) and closeness to the best possible model selection (relative accuracy). These metrics are sensible for comparing how well each technique picks the top-performing model.\n\n- The authors also assess computational overhead by measuring FLOPs, which is a standard practice for methods that claim efficiency improvements.\n\nOverall, the proposed metrics and methods are appropriate for the problem of model selection.", "theoretical_claims": "- The main theoretical contribution is an extended PAC-Bayes generalization bound that incorporates Hessians for large-scale Transformers. The bound is used to explain the emerging “pre-power” and “power” phases when fine-tuning on increasingly large datasets.\n\n- While the proofs for these claims are only summarized in the main text, the logic appears sound and consistent with prior work on PAC-Bayes bounds and Hessian-based approaches. The bounding technique used is reminiscent of standard expansions from prior bounding theorems, now customized to highlight transitions in the Hessian norm.\n- No glaring issues stand out in the conceptual extension of the Hessian-based approach, though a deeper reading of the full formal derivations (in the appendices) would be needed to confirm all details (while I didn't check very carefully). At a high level, the argument is plausible and well-motivated.", "experimental_designs_or_analyses": "- **Study design:** The authors systematically vary the amount of training data by doubling from a small subset up to a relatively large subset, documenting the test loss at each point.\n\n- **Comparisons:** They benchmark LensLLM against five methods (including strong baselines such as rectified scaling law, zero-shot performance, and subset tuning).\n\n- **Potential concerns:**\n\n  - The paper focuses on classical NLP tasks (summarization, language modeling, etc.). While these tasks are relevant, additional tasks (like natural language inference, reasoning-intensive tasks) might further validate generalization. Also, I notice that this papers' experimental design is mainly from rectified scaling law (Lin et al. 2024), while WMT19 is replaced with Wikitext, is there any concern or rationale for this?\n\n  - The discussion does not explicitly detail potential hyperparameter differences between models in the direct comparisons, although they do mention controlling for the number of epochs/steps and compute. Some clarity on controlling possible confounders (like different training schedules) would strengthen the claims.", "relation_to_broader_scientific_literature": "- This work builds on lines of research on scaling laws (e.g., Kaplan et al. 2020) and PAC-Bayes-based analyses of neural networks.\n\n- While the application is specifically targeted at large language model selection, the theoretical perspective on how Hessians and training data size interplay may also be relevant to general deep network analysis.\n\n- The approach also resonates with the established tradition of sub-model selection or partial fine-tuning to reduce compute, but it is distinguished by an explicit theoretical lens on the pre-power vs. power regime.\n\nOverall, I think this paper is heavily based on rectified scaling law (RSL). The method (the law it fit) is only different on Eq.10, where RSL uses a parameter to fit using data (namely \"pre-learned data size\"), while this paper uses a NTK-based test loss function on transformers. The theoretical analysis of phase transition is also a new contribution over RSL (while this may need an extra examination by other reviewers).", "essential_references_not_discussed": "I don't see any missing references.", "other_strengths_and_weaknesses": "- **Strengths**:\n\nThe paper offers a well-structured theoretical approach that is rare in the realm of purely empirical LLM selection.\n\n- **Weaknesses**:\n\n1. The approach might be difficult to implement for extremely large models (i.e., 30B+ parameters) unless the user has the necessary partial fine-tuning infrastructure. The authors mention partial subsets to reduce cost, but the feasibility at truly massive scales may need more real-world demonstration. The sub-fine-tuning based model selection is still costly.\n\n2. The paper could clarify hyperparameter control across different candidate models to ensure consistent comparisons (especially if some models are more sensitive to learning rate than others). \n\n3. The approach might rely on the assumption that Hessians remain relatively stable for a given model family. Future expansions might check whether or not modifications in architecture or pre-training domain strongly shift Hessian-based bounds.\n\n4. The primary concern is that the paper offers only an incremental improvement over rectified scaling law. While the proposed method achieves higher accuracy, the improvement is not substantial. The study would benefit from providing deeper insights into model selection beyond methodological refinement.", "other_comments_or_suggestions": "- It would be useful to see how robust the final ranking is under mild variations of hyperparameters or partial training steps, to confirm that results aren’t dependent on a very specific training schedule.", "questions_for_authors": "- **Phase Transition Sensitivity**: How robust is the identified transition point between pre-power and power phases to hyperparameter choices (e.g., learning rate, batch size, sequence length)? Do small changes in these settings significantly shift the transition point or degrade predictive accuracy?\n\n- **Scalability**: For extremely large models (tens of billions of parameters or more), do you expect Hessian approximations to remain stable in practice? Or is there a risk that computing these approximations or performing partial fine-tuning becomes infeasible?\n\n- **Architectural Variations**: How would a significantly modified Transformer architecture (e.g., MoE, encoder-decoder) impact lens-based predictions? Would your approach require re-fitting theoretical parameters for such architectures?\n\n- **Experimental Details**: The details of the model selection experiments are not entirely clear. What is the exact set of models used? It appears that the models tested are generally not very large—could you clarify why? Additionally, do any of the fine-tuning results come directly from the rectified scaling law, or were they trained independently? If the latter, what are the fine-tuning details, including software and hardware specifications?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DMVclSKddM", "reviewer": "Reviewer_4f9z", "summary": "The paper introduces LensLLM, a novel framework for selecting Large Language Models (LLMs) by analyzing their fine-tuning dynamics. The authors propose a Hessian-based PAC-Bayes generalization bound to model the transition phases in fine-tuning, aiming to improve the efficiency and accuracy of model selection. The work also incorporates a Neural Tangent Kernel (NTK)-based Rectified Scaling Model to predict performance across diverse tasks. Empirical results on large-scale benchmarks demonstrate LensLLM's superiority over existing methods. \n\nIt is indeed very novel, as claimed in the paper, but I really feel confused when reading the paper. Maybe I lack some important background knowledge, like that in Lin et.al 2024, but I believe a good paper should be self-contained. I would be very happy to increase my evaluation if the authors could kindly help me understand this paper better (especially the theory part).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions. The followings are our answers to your concerns.\n\nQ1: \"The analysis of the paper heavily depends on Theorem 1, which is a Generalization bound of LLM’s finetuning. I am not quite sure whether the bound is tight enough to claim that model A is better than model B since its generalization bound is smaller.\"\n\nA1: We would like to clarify that Theorem 1 is not used for direct model comparison. Instead, its primary purpose is to fit the transition phases and identify the transition point in the fine-tuning dynamics. Once this transition point is determined, we apply our regression model to predict performance and compare the predicted loss values for model selection. Thus, model selection is based on the predicted loss rather than on a smaller generalization bound.\n\nQ2: \"It is a bit hard for me to understand the definition of the feature vector x. What would x for GPT2 look like? What is the difference between x for GPT2 and T5? Plus, the notation x is also used in Assumption 1-3 in Section 3.1. Are they the same x?\"\n\nA2: \na. In our framework, $x$ represents the features from the input space, which are used to characterize the fine-tuning data. These features encapsulate relevant properties of the input samples that influence the fine-tuning process. For specific models: \n\n- For GPT-2, $x$ corresponds to features extracted from its auto-regressive training setup, where the inputs are sequences of tokens with causal masking. \n\n- For T5, $x$ represents features from its encoder-decoder input format, where the model processes full input sequences bidirectionally before generating outputs. \n\nb. The key difference is that T5 operates on a denoising objective rather than strict left-to-right token prediction like GPT-2. \n\nc. Yes, $x$ in Assumptions 1-3 refers to the same definition as for GPT-2 and T5. \n\nQ3: \"In Equation 8, I cannot understand what x and x’ represent for. Are they some features of pre-training and fine-tuning data samples? If not, and following the definition of x as aforementioned, then why do we call a transformer model f(_,_)? Do we need to provide a vector $x$ with the model size, architecture, etc., to a LLM?\"\n\nA3:\na. Yes, as we clarify just under Equation 8, $x$ and $x'$ refer to input feature representations from pretraining and fine-tuning data, respectively. \n\nb. No, we do not need to provide a vector $x$ with the model size, architecture, etc., to an LLM directly. For this part, we want to provide the details of the NTK matrix that are extracted from pre-training and fine-tuning stages, and then we will pass this information to our proposed scaling law model:\n$$\nL(D) = \\frac{B}{F(\\Theta, t) + D^\\beta} + E\n$$\nOveral, we are using the information from pre-training and fine-tuning stages of LLMs to help us find the transition pattern and do better prediction of performance. For the architecture of our model, please refer to the pseudo-code in section 3.2. \n\nQ4: \"The analysis of this paper heavily relies on the generalization bound provided in Ju et al. 2023 and the scaling law provided Lin et al 2024. But will they precisely describe LLM’s behavior?The experiments provided in Ju et al. 2023 only consider image classification tasks, which is quite different from LLM’s auto-regressive training. Compared with Lin et al. 2024, the experiments in this paper do not seem sufficient. IMO, extending this framework to LLMs’ finetuning needs more justifications.\"\n\nA4: We would like to clarify that our work fundamentally differs from (Ju et al., 2024) and (Lin et al., 2024) as follows:\n\n1. Framework difference from Ju et al. (2023): Ju et al. focus on image classification, but we incorporate transformer-specific elements—such as attention mechanisms, layer normalization, and residual connections—into the PAC-Bayesian generalization bound (as detailed in Appendix A), which are not addressed in Ju et al. \n\n2. Contribution difference from Lin et al. (2024): Lin et al. empirically capture the pre-power and power phases based on heuristic scaling laws derived from observational data. In contrast, our work rigorously verifies these phases through a theoretical framework, specifically by deriving the Hessian-based PAC-Bayes generalization bound. This theoretical foundation enables us to develop a theory-grounded scaling law that precisely describe LLM behavior.\n\nWe acknowledge that our current computational resources (using a single A100-80G) limit our ability to extend experiments to larger models, and we plan to explore this in future work. Additionally, we have conducted further tests on the robustness of our method to hyperparameters; due to space limitations, please refer to the rebuttal for Reviewer 2LaB for more details.\n\nPlease let us know if there are any comments or insights, we'd like to explore further!", "claims_and_evidence": "Not quite.\n\nThe main concern for the theoretical results of this paper is the assumptions it makes. I am not sure to what extent these assumptions could describe the transformer’s behavior. The analysis of this paper heavily relies on the generalization bound provided in Ju et.al. 2023 and the scaling law provided Lin et.al 2024. But will they precisely describe LLM’s behavior? The experiments provided in Ju et.al. 2023 only consider image classification tasks, which is quite different from LLM’s auto-regressive training. Compared with Lin et al. 2024, the experiments in this paper do not seem sufficient. IMO, extending this framework to LLMs’ finetuning needs more justifications.", "methods_and_evaluation_criteria": "Part of.\n\nThe evaluations made by this paper are similar to Lin et.al 2024, but with fewer experimental settings. However, since the proposed methods perform pretty well, the paper still has big potentials.", "theoretical_claims": "Mostly.\n\n1. The analysis of the paper heavily depends on Theorem 1, which is a Generalization bound of LLM’s finetuning. I am not quite sure whether the bound is tight enough to claim that model A is better than model B since its generalization bound is smaller.\n\n2. It is a bit hard for me to understand the definition of the feature vector x. What would x for GPT2 look like? What is the difference between x for GPT2 and T5? Plus, the notation x is also used in Assumption 1-3 in Section 3.1. Are they the same x?\n\n3. In Equation 8, I cannot understand what x and x’ represent for. Are they some features of pre-training and fine-tuning data samples? If not, and following the definition of x as aforementioned, then why do we call a transformer model f(_, _)? Do we need to provide a vector x with the model size, architecture, etc., to a LLM?", "experimental_designs_or_analyses": "Yes. The experiments look pretty good.", "relation_to_broader_scientific_literature": "The paper did a good job summarizing and discussing related works in this field.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. Theoretical Contributions: The introduction of a Hessian-based PAC-Bayes generalization bound provides a solid theoretical foundation for understanding LLM fine-tuning dynamics, although a bit hard to understand for me.\n\n2. Efficient Model Selection: The NTK-based scaling model offers a computationally efficient alternative to exhaustive fine-tuning by leveraging pre-trained model properties.\n\n3. Empirical Validation: The framework is tested on diverse datasets and multiple model architectures, demonstrating consistent improvements over many baselines.\n\n4. Open-Sourced Implementation: By providing an open-source implementation, the work promotes transparency and reproducibility in the field.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper contributes to the growing literature on efficiently guiding LLM model selection during fine-tuning. The key contributions of the paper are a PAC-Bayes generalization bound that can handle transformer architectures, and its use under an NTK model to efficiently predict the fine-tuned performance of various alternatives. This is a very timely and important topic. Reviewers found the paper to be well-written, sound, and impactful. Some of the clarifications offered during the discussions (e.g., ablation studies, architectural details, delineation from past theory) should be included in the paper. The authors are also encouraged to better discuss the limitations of their approach (e.g., scalability).
Decision: Accept (poster)