ICML 2025

LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection
Xinyue Zeng 1 Haohui Wang 1 Junhong Lin 2 Jun Wu 3 Tyler Cody 4 Dawei Zhou 1
Abstract
The proliferation of open-sourced Large Lan-
guage Models (LLMs) and diverse downstream
tasks necessitates efficient model selection, given
the impracticality of fine-tuning all candidates due
to computational constraints. Despite the recent
advances in LLM selection, a fundamental re-
search question largely remains nascent: how can
we model the dynamic behaviors of LLMs during
fine-tuning, thereby enhancing our understand-
ing of their generalization performance across
diverse downstream tasks? In this work, we pro-
pose a novel theoretical framework that provides
a proper lens to assess the generalization capabili-
ties of LLMs, thereby enabling accurate and effi-
cient LLM selection for downstream applications.
In particular, we first derive a PAC-Bayesian Gen-
eralization Bound that unveils fine-tuning dynam-
ics of LLMs and then introduce LENSLLM, a
Neural Tangent Kernel (NTK)-based Rectified
Scaling Model that enables accurate performance
predictions across diverse tasks while maintain-
ing computational efficiency. Extensive empirical
results on 3 large-scale benchmarks demonstrate
that our model achieves up to 91.1% accuracy and
reduces up to 88.5% computational cost in LLM
selection, outperforming 5 state-of-the-art meth-
ods. We open-source our proposed LENSLLM
model and corresponding results at LensLLM.io.
1. Introduction
The rise of large language models (LLMs) has revolu-
tionized natural language processing, driving remarkable
1Department of Computer Science, Virginia Tech, Blacksburg,
VA, USA. 2Department of Electrical Engineering and Computer
Science, Massachusetts Institute of Technology, Cambridge, MA,
USA. 3Department of Computer Science and Engineering, Michi-
gan State University, East Lansing, MI, USA. 4Department of In-
telligent Systems Division, Virginia Tech, Blacksburg, VA, USA..
Correspondence to...