New Hybrid Fine-Tuning Paradigm for LLMs:  Algorithm Design and Convergence Analysis Framework
Authors: Shaocong Ma
Peiran Yu
Heng Huang
Venue: ICLR 2026 Poster (2026)
Keywords: peft
large language models
fine tuning
Primary area: foundation or frontier models, including LLMs
TL;DR: We identify a smoothness mismatch in LLM fine-tuning and propose a solution to mitigate it.

Abstract: Fine-tuning Large Language Models (LLMs) typically involves either full fine-tuning, which updates all model parameters, or Parameter-Efficient Fine-Tuning (PEFT), which adjusts a small subset of parameters. However, both approaches have inherent limitations: full fine-tuning is computationally expensive, while PEFT often struggles to learn new knowledge and exhibits suboptimal performance.  To overcome these issues, we propose a novel *hybrid fine-tuning* approach that jointly updates both  LLMs and PEFT modules  using a combination of zeroth-order and first-order optimization methods. To analyze our new algorithm, we develop a theoretical framework centered on the concept of *hybrid smoothness condition*, which accounts for the heterogeneous nature of the optimization landscape in joint LLM and PEFT training. We derive a rigorous convergence analysis for the convergence of reshuffling-type SGD algorithm under multiple learning rates and demonstrate its effectiveness through extensive empirical studies across various downstream tasks and model architectures. On the practical side, our results demonstrate consistent performance improvement, making the approach a viable solution for large-scale language model fine-tuning.

Published as a conference paper at ICLR 2026
NEW HYBRID FINE-TUNING PARADIGM FOR LLMs: ALGORITHM DESIGN AND CONVERGENCE ANALYSIS FRAMEWORK
Shaocong Ma
Peiran Yu
Department of Computer Science University of Maryland College Park, MD 20742, USA scma 0 900 8 umd. edu
Department of Computer Science and Engineering University of Texas Arlington Arlington, TX 76019, USA peiran.
peiran.yu@uta.cc wwwptuuuuaaee.mmm edu
Heng Huang"
Department of Computer Science University of Maryland College Park, MD 20742, USA heng@umd edu
ABSTRACT
Fine-tuiing Large Language Models (LLMs) typically involves either full fine- tuning, which updates all model parameters, Parameter-Efficient Fine- TineiTuning (PEFT), which adjusts a small subset of parameters. However, both approaches have inherent limitations: full fine-tuning is computationally expensive, while PEFT often struggles to learn new knowledge and exhibits suboptimal perfor- mance. To overcome these issues, we propose a novel hybrid fine-tuning approach that jointly updates both LLMs and PEFT modules using a combination of zeroth- order and first-order optimization methods. To analyze our new algorithm, we develop a theoretical framework centered on the concept of hybrid smoothness condition, which accounts for the heterogencous nature of the optimization land- scape in joint LLM and PEFT training. We derive rigorous convergence analysis for the convergence rates and demonstrate its effectiveness through extensive empirical studies across various downstream tasks and model architectures. On the practical side, our re- sults demonstrate consistent performance improvement, making the approach a viable solution for large-scale language model fine-tuning.
reshufiling-type SGD algorithm under multiple learning
INTRODUCTION
Large Language Models (LLMs) have emerged as an important paradigm in natural language pro- cessing (NLP), demonstrating remarkable capabilities across wide range of tasks. To adapt these models for specific domains or to modify their core behaviors, researchers commonly employ full fine-tuning for downstream tasks (Malladi al., 2023: Zhang al., 2024; VM al., 2024; Mi- naee et al., 2024), which updates all parameters of an LLM. However, this method is extremely computationally expensive, requiring the calculation of gradients for the entire model. To address this limitation, two common approaches have introduced: (1) Zeroth-order full fine-tuming (Malladi al., 2023; Zhang et al., 2024; Gautam al., 2024: Tang et al., 2024; Wang et al., 2024; 2025): This type of methods approximates gradients without directly computing them, reducing computational overhead while still updating all model parameters. (2) Parameter-Efficien Fine-Tuning (PEFT) methods (Lester et al., 2021; Hu al., 2021; Li & Liang, 2021): These techniques aim to adapt LLMs by tuning only small portion parameters while keeping the base model frozen.
"This work was partially supported by NSF IIS 2347592, 2348169, DBI 2405416, CCF 2348306, CNS 2347617, RISE 2536663.

---

Published as a conference paper at ICLR 2026
1000
10a
4000
LogiSmoahnese
109
Lag]Smoothneess)
100
3000
5t0ps
10a
100
2000
10a
10a
1000
10
109
10
10a
10a
Log(Gradient Norm)
Base LLM
LoRA
Log-scale comparison of gradient norm to local gradient Lipschitz constant in OPT- 125M (Zhang et al., 2022) training on SST2 dataset (Socher et al., 2013). The colorbar indicates the number of gradient updates.
(h) Comparison of gradient Lipschitz con- stant for different modules (OPT-125M and LoRA (Hu et al., 2021)). The base LLM exhibits significantly larger, necessitating gradient updating.
smaller learning
Figure I: Visualization of smoothness structures in hybrid fine-tuning a large language model. These complex characteristics pose new challenges for the convergence analysis of traditional optimization algorithms, motivating us to consider a relaxed smoothness condition, hybrid smoothness condition (Definition 1), for the hybrid fine-tuning method.
However, directly applying either of these methods has been shown to be insufficient: As pointed by (Gudibande et al., 2023) and (Ghosh et al., 2024), the PEFT method (e.g. LoRA) does not learn new knowledge, while the zeroth-order full fine-tuning suffers from slow convergence due to the lack of gradient information (Nesterov & Spokoiny, 2017). These limitations highlight a critical gap in current approaches, leading to the following question:
Q1: How can we achieve both benefits of full fine-tuning and PEFT methods while maintaining the efficiency?
To address this question, we propose a novel approach, hybrid fine-tuning, which jointly updates both the PEFT module and the LLM. We integrate both first-order (FO) and zeroth-order (ZO) opti- mization techniques for conducting PEFT and updating the base model simultancously. By leverag- ing ZO methods, we can perform fine-tuning on the base LLM without calculating the full gradient, thereby effectively learning new knowledge. Meanwhile, our method updates PEFT modules using the FO gradient information, speeding up traditional ZO full fine-tuning.
To assess efficiency, we analyze the convergence of our proposed approach, which presents new theoretical challenges in the analysis. As demonstrated in existing literature (Zhang et al., 2019; Carmon et al., 2020), the optimal learning rate closely tied to the local smoothness of the loss landscape. This dependence is especially critical in setting, where the complex architecture of modern LLMs and the heterogeneous nature of hybrid fine-tuning introduce two key challenges:
a) A dynamic changing gradient Lipschitz constant. The local smoothness structure of LLMs evolves dynamically during training. This phenomenon, first observed by (Zhang et al., 2019) for LSTM-based language models, extends transformer-based architectures, underscoring the complexity of LLM fine-tuning. The Figure la illustrates this dynamic behavior in OPT-125M (Zhang et al., 2022), transformer-based language model.
b) Heterogeneous smoothness across parameters. The base LLM and PEFT modules exhibit distinct smoothness characteristics. Due to differences architecture and scale, components in our proposed hybrid fine-tuning approach naturally possess diverse smoothness properties. This heterogeneity is demonstrated in the Figure lb, which compares the gradient Lipschitz constants between the base L.M and the LoRA module.
2

---

Published as a conference paper at ICLR 2026
These challenges highlight significant gap between existing theoretical frameworks and the practi- cal implementation of hybrid fine-tuning methods: Traditional convergence analysis of optimization algorithms cannot be applicable for such complicated loss surface, which also leads to the following central question:
Q2: How develop unified theoretical framework that accurately char- acterizes the convergence of SGD for hybrid fine-tuning while accounting for their distinct characteristics and behaviors?
To answer this question, we develop a novel theoretical framework centered around the concept of hybrid smoothness condition. This framework provides more accurate characterization of the optimization landscape in joint LLM and PEFT training, enabling rigorous analysis of convergence properties and optimization dynamics. Our main contributions are summarized as follows:
(1) We propose the hybrid fine-tuning paradigm, novel approach that addresses the limitations of both full fine-tuning and traditional PEFT methods. By combining zeroth-order optimization for LLMs with first-order methods for PEFT modules, we achieve balance between adaptation power and computational efficiency. This innovative strategy further reveals the hybrid smooth- ness condition (Definition 1) of the hybrid structure, highlighting a new theoretical challenge arising from the heterogeneous structure of joint LLM and PEFT optimization.
(2) address the challenge posed by hybrid smoothness condition, we introduce unified theoreti- cal framework for analyzing hybrid optimization problems arising in hybrid fine-tuning. Within this framework, we establish the convergence of SGD with Random Reshuffling (Theorem I), addressing a previously unresolved gap in optimization theory. Notably, analysis extends the optimal sample complexity guarantees from the standard smooth loss class to the more general hybrid smooth loss function class.
(3) We conduct extensive empirical studies to evaluate the effectiveness of our hybrid fine-tuning approach across diverse set of downstream tasks and model architectures. As shown in Fig- ure 3 and Table 1, hybrid fine-tuning consistently outperforms existing methods across 18 model-task combinations (spanning three architectures and six tasks), achieving the highest accuracy in 94.5% of the cases (17 out of 18). Empirical evidence of faster convergence is fur- ther validated in Figure 4. Notably, these improvements incur additional memory overhead compared to the FO counterpart, as demonstrated in Table
HYBRID FINE-TUNING AND HYBRID SMOOTHNESS CONDITION
2.1 OUR PROPOSED METHOD: THE HYBRID FINE-TUNING
To balance the adaptation power of full fine-tuning with the efficiency of PEFT, we introduce hybrid fine-tuning, where both the base LLM and lightweight PEFT module are updated jointly.
Methodology. Our hybrid fine-tuning approach jointly updates both the PEFT module parameters and the base LLM parameters. The parameters updating tasks can be formulated as a class of optimization problems where the parameter space partitioned into distinct subspaces: Let E dh denote the LLM parameters and E IRd the PEFT module parameters, d = dy + dy For we minimize the empirical loss:
a dataset D
1 Σ f(z, y n
min (da))edd
f(x
=
(1)
In hybrid fine-tuning, we leverage ZO optimization for the parameters, which avoids computing the full gradient and thus significantly reduces memory requirements. Simultaneously, we update the much smaller PEFT module parameters (the parameter) using the FO gradient information, which leads to faster convergence and better performance compared to solely ZO methods. Our algorithm is described in Algorithm 1:
3

---

Published as a conference paper at ICLR 2026
Algorithm 1: SGD with Random Reshuffling for Hybrid Fine-Tuning
Input:
Learning rate 7 = [Nx 17y number of epochs T, dataset DD = (5 Initialize the parameter (Io, yo):
for & = to I' do
Shuffle the dataset D obtain D;;
zt,0. Vt <- -t-1, Mt-1;
for
do
0]
21,1-1
7h
Vt,
0
Ny
Vyf(ri, Ve,
end
end
Te f- Fi.n
Output: Final parameters 2T
The optimization strategy is implemented using SGD with random reshuffling, common practice in deep learning (Paszke et al., 2019; Abadi et al., 2016) demonstrated improved efficiency in ex- isting theoretical literature (Ma & Zhou, 2020; Safran & Shamir, 2020; Mishchenko et al., 2020; Gürbüzbalaban et al., 2021; Liu & Zhou, 2024). Here, 7 and nl the learning rates for and y parameters, respectively, T is the total number of epochs, D = { is the dataset with n samples, and Tt, and Yr,; are the parameter values after the i-th iteration of the 1-th epoch. Vz and of are the stochastic gradients with respect to z and y. Here, we use vz to represent the gradient estimator of VzJ. It is commonly estimated using the two-point gradient estimator defined as follows:
V Va (i f(z + uu, viE) v() U,
where e is random Gaussian vector with identity covariance matrix (that is, N(0, La)) and the perturbation stepsize.
2.2 CHALLENGES HYBRID FINE- TIN-TUNING: THE HYBRID SMOOTHNESS CONDITION
Besides intuitively designing the hybrid fine-tuning strategy, we conduct rigorous theoretical anal- ysis on the convergence of Algorithm 1. However, the convergence analysis reveals theoretical challenges stemming from the complex optimization landscape of hybrid fine-tuning. Traditional analysis often relies on the L-smoothness assumption, which states that the gradient is Lipschitz continuous with constant L, or equivalently, (²) Lla- While L-smoothness has been demonstrated to hold for all smooth functions over compact domain (Hewitt & Stromberg, 2012), this assumption is often too restrictive for deep learning models and particularly for our proposed hybrid setting. We recap these limitations we have introduced:
a) The gradient I. Lipschitz constant I. is dynamically changing during training. The constant L usually fails to maintain uniformity over the entire parameter space. In many practical scenarios, different regions of the parameter space may exhibit vastly different smoothness properties. For instance, (Zhang et al., 2019) has demonstrated that the local smoothness constant L is linear in the gradient norm. We also have illustrated this uniformffot for transformer-bat based language models in Figure la.
b) The gradient Lipschitz constant L can be different for different parameters. The base LLM (the parameter) and the PEFT module (the y parameter) inherently possess different structural properties and scales. For example, small aandomly-nitializn modules often have smaller L compared large pre-trained neural networks. This consideration becomes particularly crucial in hybrid systems where we deal with fundamentally different types of parameters. We have illustrated this point in the Figure 1b: The LoRA module demonstrates substantially lower L value compared to the base LLM.
To rigorously characterize the phenomenon observed in hybrid fine-tuning, adapt the concept of generalized smoothness (Zhang et al., 2019; Li et al., 2024) to our hybrid setting, leading to the hybrid smoothness condition:
4

---

Published as a conference paper at ICLR 2026
Definition 1 (Hybrid smoothness). A function f R x Rd + R. has the hybrid generalized smoothness property if there exist two non-negative non non-decreasing sub-quadratic functions lz Rz0 + Rg0 and fy R2 Rz0 such that for all (z, y):
l ( fzz I)
( II v
0
This definition allows the smoothness bound depend on the current gradient norm and differ between the and y parameter blocks. It naturally captures the dynamic and heterogeneous nature of the optimization landscape.
easy to see that the standard T-ssmoothness is a special case where (t) b (t) L for all As demonstrated by (Zhang et al., 2019; Li et al., 2024), many neural network loss landscapes are empirically observed to generalized smooth but not L-smooth.
The Impact Hybrid Smoothness Condition. The generalized smoothness condition presents significant challenge in training our proposed hybrid system, particularly motivating the use of two distinct learning rates. the following example of our proposed hybrid L.LM fine-tuning structure, have two sets of parameters: (I) the original LLM parameters and (2) the PEFT module parameters y. we jointly train the LLM with a Prompt Encoder (Lester et al., 2021) on the SST-2 dataset (Socher et al., 2013). We observe that the base LLM merely takes much smaller learning rate; if we choose the learning rate to ensure the base LLM's convergence, the training loss decreases in unacceptably slow (Figure 2a). However, if we choose the learning rate larger than the base L.MM's ss tolerance, the training loss explodes and quickly diverges (Figure 2b). The best practice choosing smaller learning rate for the base LLM and larger learning rate for the PEFT module (Figure 2c).
24
8az
220
1.8
1.6
206
(a) Under the small learning rate 10-6, the full fine-tuning decrease expected. However, prompt tuning converges slowly and stagnates at a higher loss.
(b) With a large learning rate
(c) Hybrid fine-tuning with dis-
7 = 10 the prompt tuning decreases expected, while full fine lin-ttning exhibits unstable be- havior, resulting in loss explo- sion.
tinct learning rates (J)= 10 for the base model, 1yy 10 for prompt tuning) provides both stable and faster convergence.
Figure 2: Comparison training loss curves under different learning rate configurations for full fine-tuning and prompt tuning on the SST-2 dataset (Socher et al., 2013) with the base model OPT- 1.3b (Zhang et al., 2022). This example illustrates the necessaity of using different learning rates in hybrid-tuning structure.
This example illustrates the practical benefits of considering hybrid smoothness condition and the resulting of different learning rates in hybrid fine-tumiie mne-tuning. is naturally to ask if this observation can be rigorously supported by the convergence analysis. We address this question in the next subsection.
2.3 THEORETICAL ANALYSIS
Recall that our objective is to solve the optimization problem presented in Eq. (1). To handle the generalized smooth structure, we introduce the following definition:
Definition 2 (Coercive). A continuous function f Rd + is coercive if the sub-level set fo e Rd fxx a) is compact for all a R.
5

---

Published as a conference paper at ICLR 2026
In the existing literature of generalized smoothness (Li al., 2024), this assumption is usually replaced with an equivalent statement: the objective function f(z, y) tends to positive infinity when (I, y) approaches the boundary of its domain. We make the following standard assumptions to regularize the function class and subsequently provide the non-asymptotic convergence analysis.
Assumption 1 (Regularity Conditions). The objective function f( v) = E ((,,1;)) de- fined in Eq. (1) satisfies the following conditions:
(1) f()) is coercive.
(2) f() is bounded below by f := (a.))))) f(, v) -00.
(3) f()) and each individual loss function (() are twice continuously differentiable.
These regularity conditions are essential for several reasons: Coercivity prevents the optimization process from diverging too far. The lower bound guarantees that the optimization problem is well- posed. Twice continuous differentiability allows for the application of various optimization tech- niques and facilitates theoretical analysis. All of them are standard and widely used in the optimiza- tion literature (Li et al., 2024).
Assumption (Bounded Variance). There exists such that for all E R
[Vf((,y))) - vf(x,)))1 0
This bounded variance assumption is standard in the analysis of reshuffling-type SGD. We note that this assumption could be further weakened to the expected smoothness (Mishchenko et al., 2020; Khaled & Richtárik, 2020). We maintain the current version for the simplicity.
With both assumptions in place, we analyze the complexity of Algorithm under the hybrid smooth- ness condition (Definition 1). Our main theoretical result summarized in the following theorem:
Theorem 1. Suppose that Assumption I and Assumption 2 hold the objective function f(z y) ==
1 - y; i), with satisfying the hybrid smoothness condition (Definition 1). Let ((r,, Yt yt)r 1T be the SGD dynamic generated by Algorithm I for solving the optimization problem Eq. (1). Let learning rates na, fy be chosen as
The min { 0(
(OLLEnd,
1
My S min (O
Lig O0 VTTLL,
Lyp VT
and the perturbation stepsize ps and the smoothness characteristies of the and y parameters Le, Lys Lu, Ly.max are specified in the appendix. Let 8 E (0,1). If the maximum number then with the probability least 1 1- 5,
of epoch T chosen as 2 0(
Σ EIV (1V,(()))))) 81)1 < 2. KT
T
Given that each epoch processes data points, the total gradient complexity is n7 O0 +e-)) This result is optimal when e sufficiently small, aligning with the best-known upper bounds es- tablished in previous convergence analyses for both generalized smooth non-convex objectives (Li et al., 2024; Zhang et al., 2019) and L-smooth non-convex objectives (Mishchenko et al., 2020; Khaled & Richtárik, 2020). Importantly, also matches the known lower bound for the SGD algo- rithm (Arjevani et al., 2023), further confirming its optimality.
Remark. On the theoretical side, our analysis highlights the asymmetry between the learning rates 7 and fyy which arises from the distinct smoothness properties of each variable. This result emphasizes the necessity of adopting tailored learning rate schedules when optimizing modules with hybrid smoothness, an aspect not addressed in standard SGD analysis, validating empirical observation in Section 2.2. Furthermore, to the best of our knowledge, there is no prior work in the optimization literature that investigates optimization methods under generalized smoothness while accounting for random reshuffling. Our results constitute the first convergence analysis in this setting.

---

Published as a conference paper at ICLR 2026
Prompt
Prefix
LoRA
100
90
Accuracy
80
60
40
SST-2
RTE
WSC
WIC
COM WnoG.
55T-2
ITE
WSC
WIC
COPA WinoG.
55T-2
WSC
WiC
COM
WroG.
ZO-Shot
FO-PEFT
Hybrid
FO-FT
Figure 3: Comparison among Hybrid Fine-Tuning Tuning (Hybrid), FO PEFT methods (FO-PEFT), FO full fine-tuning (FO-FT), and ZO full fine-tuning (ZO-FT). In 13/18 zs 72.2% combinations, Hybrid Fine-Tuning outperforms both ZO and FO full fine-tuning.
EXPERIMENTS
Following similar setting of ZO-Bench (Zhang et al., 2024), we evaluate the performance of our proposed method on six representative datasets using three different LLMss Experimental results show that our proposed method consistently achieves superior performance and faster convergence.
Tasks & Datasets. We assessed our approach on 6 representative NLP tasks including the sen- timent classification task the SST2 dataset Socher al. (2013), the sentence differing task the WSC dataset Levesque et al. (2012), contextualized word and sense representation and word sense disambiguation task on the WiC dataset Pilehvar & Camacho-Collados (2018), the question answering task on the COPA dataset Roemmele et al. 1), and the common sense reasoning task on the WinoGrande dataset Sakaguchi al. (2021).
Experimental Setting.
Following the methodology of Malladi et al. (2023); Zhang et al. (2024), assessed our approach on the SST2 dataset Socher et al. (2013), the sentence differing task on the WSC dataset Levesque et al. (2012), contextualized word and sense representation and word sense disambiguation task on the WiC dataset Pilehvar & Camacho-Collados (2018), the question answering task on the COPA dataset Roemmele al. (2011), and the common sense reasoning task the WinoGraande dataset Sakaguchi et al. (2021). The models we use in our experiments include OPT-1.3b Zhang et al. (2022), Vicuna-7b Chiang et al. (2023), and LLaMA-7b Zhang al. (2023b). We compare the per- formance of our approach against standard PEFT methods including first-order prompt tuning Lester al. (2021), LoRA tuning Hu al. (2021), and prefix tuning Li & Liang (2021). For each dataset, we randomly sample 1,000 examples for training, 1,000 examples for evaluation, and 100 examples for development. Performance evaluated using accuracy All experiments utilize SGD as the optimizer. In the case of prompt tuning and prefix tuning, the prompts are initialized according to the predefined settings Table E.2 of Malladi et al. (2023), while for LoRA tuning, initialize with zeros. We perform hyperparameter tuning for all methods and report the best configurations. To ensure fair comparison, we keep the cardinality of the hy- perparameter search spaces identical. We set the maximum number of training steps 20,000, with early stopping applied when applicable. The detailed hyperparameter setting, overviews of the tasks and PEFT methods, paramerrr setting, and the full results are reported in the supplementary materials.
representative NLP tasks including the sentiment classification task
score, as appropriate for each task.
3.1 COMPARISON WITH FO AND ZO FULL FINE-TUNING
We begin by evaluating our method on the medium sized OPT-1.3b model to assess the performance gains over both FO and 70 full-parameter fine-tuning.

---

Published as a conference paper at ICLR 2026
Table 1: Experiment results for various fine-tuning methods applied to three large language mod- els across six NLP tasks. Highlighted cells denote the best score for each comparision pair. The left panel (Pairwise Comparison) presents side-by-side comparisons of each Hybrid variant with its corresponding first-order (FO) method (e.g. FO-Prompt
Hybrid-Prompt), enabling direct per- formance comparisons. The Hybrid method outperforms its FO counterpart in 41 out of 54 cases ( 76%). The right panel (First-Order PEFT vs. Hybrid) groups all FO-based methods (Prompt, Prefix, LoRA) separately from their Hybrid counterparts, emphasizing the overall gains from hybrid fine-tuning, In 17 out of 18 comparisons (~e 94.5%), Hybrid variants yield superior performance.
Pairwise Comparison
First-Order PEFT Hybrid
Task
SST-2 RTE WSC WiC COPA WinoG.
Task Task Type
SST-2
RTE WSC WiC COPA WinoG. Classffiation
Model
Took Pype
Classification
Reasoning
Reasoning
FO-Prompt
95.6
59.9
36.5
58.5
88.0
67.2
FO-Prompt
95.6
59.9
36.5
58.5
88.0
67.2
Vieuna- Vicuna-7b-vi. 7b-1.5 Llama-2-7h
Hybrid-Promph
95.9
59,9
61.5
64,4
88.0
68.9
FO-Prefix
91.1
60.6
51.9
51.7
83.0
66.2
FO-Prefix
91.1
60.6
51.9
51.7
83.0
66.2
FO-LoRA
94.6
62.1
60.6
61.6
84.0
68.5
Hybrid-Prefix
91.6
60.6
42.3
51.5
85.0
64.3
Hybrid-Prompt
95.9
59.9
61.5
64.4
88.0
68,9
FO-LaRA
94.6
62.1
60.6
61.6
84.0
68.5
Hrbrid-Prefix
91.6
60.6
42.3
51.5
85.0
64.3
Hybrid-LoRA
93.4
62.5
60.6
61.7
88.0
66.3
Hybrid-LoRA
93.4
62.5
60.6
61.7
88.0
66.3
FO-Prompt
94.4
82.3
64.4
61.0
84.0
65.8
FO-Prompt
94.4
82.3
64.4
61.0
84.0
65.8
Prompp
95.0
70.1
55.8
64.7
84.0
66.3
FO Prefix
90.0
70.4
61.5
56.6
80.0
64.1
FO-Prefix
90.0
70.4
61.5
56.6
80.0
64.1
FO-LoRA
94.6
80.1
53.8
58.5
85.0
66.7
Hybrid-Prefix
90.7
80.9
52.4
83.0
74.0
Hybrid-Prompi
95.0
70.1
55.8
64.7
84.0
66.3
FO-LoRA
94.6
80.1
53.8
58.5
85.0
66.7
Hybrid-Prefix
90.7
80.9
66.3
52.4
83.0
74.0
Hybrid-LoRA
92.2
82.0
72.1
66.8
84.0
66.7
Hybrid-LoRA
92.2
82.0
72.1
66.8
84.0
66.7
FO Prompt
91.3
52.3
44.2
57.5
74.0
$7.8
FO Prompt
91.3
52.3
44.2
57.5
74.0
$7.8
OPT-1.3b
Hromid
91.7
62.5
57.7
63.3
77.0
59.9
FO-Prefix
92.2
48.3
61.5
52.5
77.0
58.3
FO-Prefix
92.2
48.3
61.5
52.5
77.0
58.3
FO-LoRA
92.2
54.8
$9.6
52.5
78.0
59.0
Hrbrid-Prefix
91.7
52.7
55.8
52.7
78.0
60.0
Hybrid-Prompt
91.7
62.5
57.7
63.3
77.0
59.9
FO-LORA
92.2
54.8
59.6
52.5
78.0
59.0
Hybeid-Prefix
91.7
52.7
55.8
52.7
78.0
60.0
Hybrid-LoRA
92.3
61.0
55.5
78.0
58.3
Hybrid-LoRA
92.3
61.0
61.5
55.5
78.0
58.3
Results. We apply our proposed hybrid fine-tuning method to six benchmark tasks using the OPT- 1.3b model. As shown in Figure 3, hybrid fine-tuning outperforms its corresponding FO-PEFT counterpart, as well as both FO and ZO full fine-tuning in most scenarios. For example, when using the Prompt Encoder as the PEFT module, hybrid fine-tuning consistently achieves the highest performance across all six tasks, demonstrating robust improvements over all baseline approaches.
3.2 PERFORMANCE ON LARGE LANGUAGE MODELS FINE- TUNNNG
Here we conduct extensive experiments to evaluate the effectiveness of our proposed hybrid fine- tuning approach.
Results. In Table we present the comparison between the proposed hybrid fine-tuning and its corresponding FO PEFT fine-tuning. In the aggregate view (right panel), hybrid tuning outperforms FO-based PEFT in 17 out of 18 cases (94.5%), demonstrating its consistent advantage. In the pairwise comparison setting (left panel), where each FO PEFT method is compared with its hybrid variant across three models and six tasks, the hybrid fine-tuning achieves better performance in 41 out of 54 combinations (=e 76%). These results underscore the effectiieness of hybrid fine-tuning, highlighting its potential as a more robust strategy for adapting LLMs to diverse downstream tasks.
3.3 VISUALIZATION OF THE IMPROVED CONVERGENCE RATE
To verify that hybrid fine-tuning converges faster than other methods, we present the training curves (including the training loss, validation accuracy, and the test accuracy) for OPT-1.3B Zhang et al. (2022) model on SST-2 Socher al. (2013) dataset in Figure 4.
Results. We observe that a significant efficiency gain in terms of training steps. For example, as shown in Figure 4. the hybrid fine-tuning takes around 2,500 steps to achieve 90% accuracy, while other methods require at least 12,500 steps to reach the same accuracy. This trend is observed across different tasks, PEFT methods, and model architectures, suggesting that the efficiency of
8

---

Published as a conference paper at ICLR 2026
hybrid tuning scales well (e.g. for Vicuna-7b-vl., model the WinoGrande dataset provided in the supplementary materials).
Train
Validation Accuracy
Test Accuracy
Hybrid Tuning
0.8
ZO-FT
0.90
0.90
FD Prompr
0.85
0.85
0.80
Train Lass
0.80
0.75
0.4
0.75
0.70
0.70
0.65
0.2
Hybrid Tuning
Hybrid Tuniin FO-Prompt Z0-FT
0.65
FO Prompt
0.60
ZO-FT
0.60
0.55
5000
10000 Steps
15000
20000
5000
10000 Steps
15000
20000
5000
10000
15000
20000
Steps
Figure 4: Training curves for OPT-1.3B model with the prompt tuning the SST2 dataset. The Hybrid Fine-Tuning method achieves significantly faster convergence than the other two baselines.
Table 2: Comparison of theoretical, asymptotical, and empirical memory usage for different fine- tuning optimizers. |x| and |y] denote the parameter counts of the base LLM and the PEFT module, respectively, with y// |>|| 0. |a| and |b] are the per-layer gradient states kept during optimization.
Optimizer
Theoretical Memory
Asymptotical Memory Empirical Memory Consumed GPU
FO-SGD (LLM)
Σ max lael. [2rl} lar
max larl, [mzl] Σ
54 GB
2xA60000
70-SGD
max, [ing
max, [utel
GB
1xA6000
(LLM)
FO-SGD (Prompt)
Σ max{ [bel. [Bel + [a]]
[x]
GB
1xA6000
Hybrid ZO-SGD (LLM+Prompt)
Σ maxe [uel) + max (zxel (u]
[a]]
46 GB
1xA6000
3.4 MEMORY USAGE ANALYSIS
In this section, we consider the memory usage the hybrid fine-tuning approach. Let [ and lyd denote the parameter sizes of the f-th layer in the base L.LM and the PEFT module, respectively, with |x| :1==;> Ee | > yy| = Ee [ye] in most practical scenarios. During first-order optimization, each computational graph node stores local gradient states, represented as aş for the LLM and for the PEFT module. A key observation is that despite updating additional parameters with the inclusion of both the base LLM and the PEFT module, the hybrid fine-tuning approach does not increase the asymptotical memory usage (i.c. as
|>] + 0). While the thcoretical memory footprint of Hybrid ZO- SGD (LLM+PEFT) is |ar| + ly], it remains dominated by |ær| in practice. Thus, the hybrid fine-tuning method enables updating more parameters without sigmificantly increasing memory consumption, ensuring scalability even for large-scale LLMs.
[y]
Results. Furthermore, our empirical results confirm this observation. Table 2 reports both the the- oretical memory requirements of several fine-tuning strategies and their actual peak GPU memory usage when fine-tuning Llama-2-7B on the SST-2 dataset. The hybrid approach not only signif- icantly reduces memory overhead compared FO full fine-tuning (+= 15% reduction), but also matches the memory footprint of FO prompt tuning (Lester et al., 2021).
3.5 EXTENDED COMPARISON OF GRADIENT LIPSCHITZ CONSTANT
In this subsection, we present extended experiments to further examine the local geometry the op- timization landscape. Specifically, we directly estimate and compare the gradient Lipschitz constants

---

Published as a conference paper at ICLR 2026
10
10
10
10
10
10
10'
10
10
10P
Extended comparison of gradient Lipschitz constant for OPT-1.3b (Left) and LLaMa- 2-7b (Right) in different modules (Base LLM and Prefix Tuning). The base LLM exhibits signifi- cantly larger Lipschitz constant, further confirming our observation in Figure 1b.
Figure 5:
Table Empirical memory us- age of different optimizers for the OPT-1.3b model. We note that the Adam optimizer takes three-time memory of the SGD optimizer due to the need to store two addi- tional state variables (the first and second moments) for each model parameter.
Table 3: Pairwise comparison between FO and Hybrid vari- ants OPT-1.3b across six NLP tasks. Additionally pared to Table 1, we include the Adam optimizer as the base- line. Notably, our proposed method still achieves advanced performance without adopting the Adam optimizer. In prin- ciple, our approach can be further enhanced by replacing the SGD update for the PEFT module with Adam to further ac- celerate the training.
Optimizer
Emp. Mem.
Model
Task
SST-2
RTE
WSC
WiC
COPA
WinoG.
FO-SGD
11.2 GB
(LLM)
FO-LoRA (Adam)
OPT-1.3b
91.7
58.6
58.7
64.1
66.0
60.1
ZO-SGD
FO-LoRA
(SGD)
92.2
54,8
59.6
52.5
78.0
59.0
(L.LM)
6.8
Hybrid-LoRA
FO-Adam (LoRA)
(SGD)
92.3
61.0
61.5
55.5
78.0
58.3
11.0 GB
Hybrid SGD (I..M+L.oRA)
10.7 GB
associated with the base model parameters (2:) and the PEFT parameters (y) variables across multi- ple models (OPT-1.3b and LLaMa-2-7b), As shown in Figure 5, these additional results consistently support our hybrid smoothness assumption by showing that the x=coordinates exhibit noticeably larger Lipschitz constants, indicating the necessity of applying different learning rate scale.
3.6 EXTENDED COMPARISON WITH THE ADAM+LORA BASELINE
In our previous comparison, using SGD across all methods was driven by our theoretical focus; the core of our contribution is the convergence analysis for our hybrid method under the novel Hybrid Smoothness condition, which we developed specifically for SGD with Random Reshuffling.
To further validate the effectiveness of our approach, we conduct an additional set of experiments comparing Hybrid-LoRA with the Adam+LoRA baseline. This extended evaluation examines whether the performance gains observed in our main results persist under the widely applied op- timization settings. The results reported in Table demonstrate that Hybrid-LoRA still maintains its advantage while using smaller memory (as indicated by Table 4). We also emphasize that our framework is extensible, and one could construct a "Hybrid-Aue -ddmm"- variant by replacing the SGD update for the PEFT module with Adam. We view this interesting direction for future work.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. (TensorFlow): system for (Large-Scale) machine learning, In 12th USENIX symposium operating systems design and implementation (OSDI 16), pp. 265-283, 2016.
Yossi Arjevani, Yair Carmon, John C Duchi, Dylan Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199 214, 2023.
Eric Buehler and Markus Buehler. X-lora: Mixture of low-rank adapter experts, a flexible frame- work for large language models with applications in protein mechanics and molecular design. APL Machine Learning, 2(2), 2024.
Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points i. Mathematical Programming, 184(1):71-120,, 2020.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna, Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Alexander Gasnikov, Anton Novitskii, Vasilii Novitskii, Farshed Abdukhakimov. Dmitry Kamzolov, Aleksandr Beznosikov, Martin Takáč, Pavel Dvurechensky, and Bin Gu. The power of first-order smooth optimization for black-box non-smooth problems. arXiv preprint arXiv:2201. 12289, 2022.
Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance- reduced zeroth-order methods for fine-tuning language models. arXiv preprint arXiv:2404.08080, 2024.
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, S Ramaneswaran, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, and Dinesh Manocha. A closer look at the limitations of instruction tuning. In Forty-first International Conference on Machine Learning. 2024.
Arnay Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary 1lms. arXiv preprint wrXi5:2305.15717 2023.
Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, et al. Zeroth-order fine-tuning of Ilms with extreme sparsity. arXiv preprint arXiv:2406. 02913,, 2024.
Mert Gürbüzzaalaaan Asu Ozdaglar, and Pablo A Parrilo. Why random reshuffling beats stochastic gradient descent. Mathematical Programming, 186:49-6 -4, 2021.
Edwin Hewitt and Karl Stromberg. Real and abstract analysis: modern treatment of the theory of functions of a real variable. Springer Science Business Media, 2012.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 10XY::2006.09685.. 2021.
Nam Hycon- Hyo,n-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efffieen federated learning. arXiv preprint arXiv:2108.0660 06098, 2021.
Ahmed Khaled and Peter Richtárik. Better theory for sgd in the nonconvex world. arXiv preprint 00X2::2002.03329, 2020.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parfmeter-efffcient prompt tuning. arXiv preprint arXiv:2 104 104.08691, 2021.

---

Published as a conference paper at ICLR 2026
Hector Levesque, Emest Davis, and Leora Morgenstem. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning, 2012.
Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. Convex and non-convex optimization under generalized smoothness. Advances in Neural Information Processing Systems, 36, 2024.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint wrxiv:2111.00/11 2021.
Zhenqing Ling, Daoyuan Chen, Liuyi Yao, Yaliang Li, and Ying Shen. On the convergence of zeroth-order federated tuning for large language models. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 1827-1838 2024.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning, Advances Neural Information Processing Systems, 35:1950-1965, 2022.
Liu Liu, Minhao Cheng, Cho-Jui Hsich, and Dacheng Tao. Stochastic zeroth- optimization via variance reduction method. arXiv preprint arXiv: /805. 181 2018.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P- tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07660 2021.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 2023.
Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsich, and Yang You. Sparse mezo: Less parameters for better performance zeroth-order Ilm fine-tuning. arXiv preprint 1rX1v:2402.15751, 2024.
Zijian Liu and Zhengyuan Zhou, On the last-iterate convergence of shuffling gradient methods. arXiv preprint ar arXiv:2403.07723. 2024.
Shaocong Ma and Yi Zhou. Understanding the impact of model incoherence on convergence of incremental sgd with random reshuffle. In International Conference Machine Learming, pp- 6565-6574. PMLR, 2020.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038-53075. 2023.
Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Am- atriain, and Jianfeng Gao. Large language models: A survey. arXiv preprint urXiv:2402.06196, 2024.
Konstantin Mishchenko, Ahmed Khaled, and Peter Richtárik. Random reshuffling: Simple analysis with vast improvements. Advances in Neural Information Processing Systems, 33:17309-17320, 2020.
Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527-566, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Mohammad Taher Pilchvar and Jose Camacho-Collados. Wic: the word-in-context dataset for eval- uating sentext-sensitive meaning representations. arXiv preprint arXiv:1808. 091211 2018
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 20/ AAAI spring symposium series, 2011 1.
12

---

Published as a conference paper at ICLR 2026
Itay Safran and Ohad Shamir. How good is sgd with random shuffling? In Conference Learning Theory, pp. 3250-3284. PMLR, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande; An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642,201 2013.
Xinyu Tang, Ashwinee Panda, Milad Nasr, Saeed Mahloujifar, and Prateek Mittal. Pri- vate fine-tuning of large language models with zeroth-order optimization. arXiv preprint arXiv:2401.04343 2024.
Kushala VM. Harikrishna Warrier, Yogesh Gupta, al. Fine tuning Ilm for enterprise: Practical guidelines and recommendations. arXiv preprint orXiv:2404.10779 2024.
Fei Wang, Shen, Liang Ding, Chao Xue, Ye Liu, and Changxing Ding. Simultaneous computation and memory efficient zeroth-order optimizer for fine-tuning large language models. arXiv preprint arXiv:2410.09823, 2024.
Liangyu Wang. Jie Ren, Hang Xu, Junxiao Wang, Huanyi Xie, David E Keyes, and Di Wang. Zo2: Scalable zeroth-order fine-tuning for extremely large language models with limited gpu memory. arXiv preprint 2rXiv:2503./22 /2668, 2025.
Shih- Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating rext-to-image customization: From lycoris fine-tuning to model evaluation. In The Twelfth International Conference on Learning Representations, 2023.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv preprint 190.v:190. 2019.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. arXiv preprint arXiv:2303. 10512, 2023a.
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero- init attention. arXiv preprint arXiv:2303.16199 2023b.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 20Xv:2205.00068, 2022.
Yihua Zhang, Pingzhi Li. Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenging Zheng, Pin- Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient Ilm fine-tuning: A benchmark. arXiv preprint arXiv:2402.1 1592, 2024.
Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W Tsang. Second-order fine-tuning without pain for 1lms: A hessian informed zeroth-order optimizer. arXiv preprint arXiv:240..1117, 2024.
13

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Zeroth-Order Optimization in Fine-Tuning L.Mss Recent work has explored ZO optimization methods for fine-tuning LLMs, which aligns with our approach of using ZO methods for the LLM component in hybrid fine-tuning. Malladi et al. (2023) demonstrated the compatibility of zeroth- order methods with both full fine-tuning and PEFTs. This laid the groundwork for our hybrid ap- proach that combines zeroth-order LLM updates with first-order PEFT updates. Zhang et al. (2024) provided comprchensive benchmark for ZO optimization in LLM fine-tuning, offering valuable insights that informed our experimental design. Ling et al. (2024) combines the ZO fine-tuning of LLMs with the federated learning. Several studies have incorporated variance reduction techniques (Gautam et al., 2024) into ZO methods or second-order method (Zhao et al., 2024) to enhance sta- bility and convergence in fine-tuning LLMs. While we focus on different aspect, these stability improvements could easily be integrated into our hybrid framework. Existing literature (Liu et al., 2024; Guo al., 2024; Zhang et al., 2024) also discusses the sparsity of pre-trained LLMs, which further enhances the performance of ZO optimization approach.
Generalized Smoothness of Large Machine Learning Models The concept of generalized smoothness has emerged as a crucial theoretical framework for understanding the optimization land- scape of large machine learning models, including L.LMs. Recent studies have shown that traditional smoothness assumptions often fail to capture the complex optimization landscape of deep neural net- works (Zhang et al., 2019; Li et al., 2024). More explicitly, Zhang et al. (2019) demonstrated that the local smoothness constant in neural networks is often proportional to the gradient norm, challeng- ing the conventional assumption of uniform smoothness. This insight aligns with our observations in hybrid fine-tuning, where different components of the model (LLM and PEFT modules) exhibit distinct smoothness properties. Li et al. (2024) introduced a generalized smoothness condition that allows for non-uniform smoothness across the parameter space, which is more representative of the behavior observed in practice for large models. This work provides a foundation for our hybrid gen- eralized smoothness framework, which extends these ideas account for the heterogeneous nature of joint LLM and PEFT optimization.
B NOTATIONS
In this paper, the optimization problem is formulated as minimizing f(z, y). where Rd repre- sents the parameters of the base language model and y E R4 represents the parameters of the PEFT module. The function f is assumed to have hybrid generalized smoothness, characterized by non- negative, non-decreasing sub-quadratic functions fg and by (Definition 1). In the SGD, we consider epoch-wist wise optimization algorithm described Algorithm 1. This approach ensures us to access each data point exactly once over an entire epoch, which is particularly common is the data loader provided by existing modern machine learning frameworks such as PyTorch and TensorFlow. Here, 7 and ny denote the learning rates for and respectively, T is the number of epochs, and is the dataset size. We Vz denote the zeroth-order gradient estimator for æ, while Vg represents the standard gradient for With these given, for each epoch t, we define the following notations:
9t
V v ( Ve..., Ee,,, δ Va.ii..
h4 V Vaf( (1.
Here, St. represents the true gradient with respect to accumulated over an entire epoch. It captures the overall direction of stochastic gradient descent for the r parameters across all samples in the often don't "on't have access to the true gradient
epoch. ğe is an estimate of this gradient. In practice, and must rely on estimates. The difference between St and g. quantifies the estimation error in our gradient calculations. by is the true gradient with respect y accumulated over the epoch.
14

---

Published as a conference paper at ICLR 2026
C SUPPORTING LEMMAS
In this section, we present several lemmas used to build our convergence analysis. Lemma 1, Lemma 2, Lemma 3, and Lemma 4 are fundamental properties of generalized smoothness provided by Li et al. (2024). We adapt them to the setting of hybrid system fine-tuning.
Lemma (The generalized version of Lemma 3.3 from Li et al. (2024)), Let f: R = Rd xI IR be a twice continuously differentiable function satisfying the hybrid generalized smoothness prop- erties. Suppose that (a., y/ R satisfies [ (x. v)ll I G. Then there exist non-negative constant Lg (G) and Ly = (G( such that for all (a1. 91), (22, U2) E B(x, I.,) Bly i.):
1. [Vz((11 5/) V=f(22, f/ L L= zorl, all v/ E R
2. W .n) y (x', u2)ll S Tyllvv for all x! TRd=.
3. Let la represent the identity matrix with the size d.
f(z1,30 v1) Sf(22, 1/2) +
f( 8/2),
21 - 322
91
82
+ 2 |
x2
y1
y/2
Lald=
Lyyld,
0
21 22
y/1
1/2
Proof. Let (zr, y) I2 = R x I be arbitrary. By the assumption of twice continuous differen- tiability and the mean value theorem, we have
/ fan t(22 () xr dtt.
V vzf(22. V=f(=i.V)
Taking the norm of both sides and applying the generalized smoothness of f (Definition 1), we obtain
V w)1l S (< v)l) L.
where the last inequality is by the monotonicity of Ea and the bounded gradient condition. We apply this inequality to the integral yields the first inequality. The second inequality for the y-gradient obtained similarly. For the third inequality, still consider the mean value theorem:
f(1 v1) f(I2, v2) = L.'
V v)(((1),
dt
9/
1/2
(122, 82);
I11
x2
V)(=(()) V U2),
21 T2
dt
1/1
122
y1 y2
201 222
+ / (Vf(<(t)) - v. 82),
x1 02
ff(22 82).
dt
1/1
1/2
// 1/2
V fx(22 v2),
21 222
+ Lyllu1 - 22l2 / tdt + L - x2l1 telt,
1/1 1/2
where z(t) := (1 t)
222
+
x1
for < $ 1. Then the proof is completed by re-arranging
1/2
y1
this inequality.
Lemma (The generalized version of Lemma 3.5 from Li al. (2024)). Let J : Rdds x Rdd, + IK be twice continuously differentiable function satisfying the hybrid generalized smoothness properties. Let 8 inf f(x, y) be the global minimum of J. Then, for all (x, y) E TRdz x R&,, the following inequalities hold:
1. V V2f(z, 1)1 22, v) (f( f*)
2. W,, 12 < (2] x)l (f(x,y) fr)
3. vv)
E (.
([(,,)){$(e. - -
15

---

Published as a conference paper at ICLR 2026
Proof. The first and the second inequalities are directly implied by Lemma 3.5 from Li et al. (2024)
by projecting the objective function f to a subspace of the domain. Here, we provide the proof for the third inequality. By Lemma I where choose G [VV y)ll. we have that for any (I1, B1), (22, B(x, G x Bly, f.).
Tiz
I1 -22
L
0 Lygdd
x1 32
M() S f(22, + (222 v2),
+ [1
22
y1 y2l
91 1/2
y/ 1/2
Choosing (x2, = (z, 1), 11 = x
v.f(2.g) /
and y! = y
V (Z
we obtain
V v=)(x.V)
V fz, v)
f S f(x
( (z. 3)
1 $ f(x,)) 2 [f(
((..))
vf(z.y).
bay (
Then the proof is completed.
Lemma 3 (The generalized version of Corollary 3.6 from et al. (2024)). Let R x R + RR be a twice continuously differentiable function satisfying the hybrid generalized smoothness proper- ties. Suppose that [xx, u) - J S F for some (ar, y) E IR and F 2 0. Denoling G == sup R 2 0
u² < 2 22max (lz, fyy (u) F), then [ y) S < oc.
Proof. Let max(ez, (u) max (2 (u), (u)) Since both lg and by are sub uub-quadratie, it con- cludes G is finite (by Corollary 3.6 from Li et al. (2024)). From Lemma 2, we have
fz. B)]
max (2) (
Idz
0
Vf(x,)) y)
2
0
Fag max 12 V (x,))))))
Lae ((m)i)
1
V fx
Ia
2
0
7, V(
$f(z,v)
Therefore, we obtain
f v)ll < 2 max(La, (yVV v)) F.
It concludes that if the function value is bounded, then the gradient is also bounded.
Here, we summarize the previous results in the following lemma. The constant (determined by the function value upper bound F) is defined in Lemma 3 and the constant La and Ly (determined by the gradient norm upper bound G) defined in Lemma 1.
Lemma 4. Suppose that Assumption 1 holds for the objective function f(x,))
1 +E= f(x, i), with all individual loss functions ((5 are twice continuously differentiable and satisfy the hybrid generalized smoothness properties. Let Gr := ((x,V) Rdd [(x - 8 F). Then the following statements hold:
1. The objective function f(-) has G-bounded gradient over Gr; that is, |V S G all (x.y) E Gr.
2. The objective function f()) has (Le, Ly) -Lipschit gradient over Gp: that is, f y) fuf g)ll Le o z'll and [VV,f(r,) v) (x, (x,z/)) S Lylly s'll for all
(x, v), (x', .') Gr.
3. The individual loss function (c; has (Gn,max: Fyymmxx)bouunde gradient over Gr; that is. 1 E)ll S Ga,max and
Elll S Gy,max for all (x,2) E Gr and all
4. The individual loss function f(:;)) has (Ln Ly.max) ) -iisschitz gradient over Gp: that is, [VV y) - V.,(2', (2', yl I Ln,max x| x' and || V y:
V. f (x)ll S Tyy.maxlly - y for all (I, v) E Gr and all g (1.
16

---

Published as a conference paper at ICLR 2026
Proof. By Assumption 9F is compact set. By the twice continuous differentiability of the ob- jective function f()) (and all individual loss functions f(=:)), all statements holds by its continuity. More precise evaluation is given in Lemma 1 for Lg and Ly. and in Lemma for G.
The following lemma characterizes the accuracy of zeroth-order gradient estimation. We note that the choice of zeroth-order gradient estimator is the crucial part in our analysis; the following gradient estimation method can be replaced with any common zeroth-order optimization techniques, including the mini-batch zeroth-order gradient estimation Nesterov & Spokoiny (2017), the uniform smoothing Gasnikov et al. (2022), and the variance reduction Liu et al. (2018).
Lemma 5. Let f Rd + IR be a function with twice continuous differentiability. Define the point zeroth-order gradient estimator of V f(x)) as
1 [f vo) f()) μ
f(z))
where μ 0 is the perturbation stepsize, v E R is Gaussian vector with the covariance matrix Ia. Suppose that f has G-bounded gradient and L-Lipschitz gradient at a. Then
1. Egg. fz()) - vf(()) - 5
for any g R
2. E f(x) vf(2)] - 32d +
Proof. Throughout this proof, we follow the random gradient-free oracles given by Nesterov & Spokoiny (2017). That is, define
(r( (0 + pv);
then the gradient estimator fx is an unbiased estimator of fz()). For the first inequality, we have
Egg. fx()) (x=)) Eig. (.a()) - Vf(z))
L(d //
where (i) applies the unbiasedness of Gaussian smoothing and (ii) applies Lemma 3 from Nesterov & Spokoiny (2017). For the second inequality, we have
H f(x) - 2 < 2E (z)) + + [x))]
4) +
32d V 108
where (i) applies Lemma 5 from Nesterov & Spokoiny (2017) and (ii) again applies Lemma from Nesterov & Spokoiny (2017).
Lemma 6. Suppose that Assumption I and Assumption hold for the objective function f(x y)
> [x,,y; i), with all individual loss functions ((3 are twice continuously differentiable and satisfy the hybrid generalized smoothness properlies. Let
E, 1 --- ff Ve.s: Ee,i)
n
(e,,,, Vf(ce, vc).
TI
be the gradient approximation error over the 1-th epoch. Given any F. H > O, define the stopping time as T = T1 A T2. where Tı == min {t f(1t+1, (1-1))++)) -5 > F} AT and T2 = min, (t leell НYTT. Let the learning rates satisfy nh min 2aa. 384L.nd. } and fy 2
and the perturbation slepsize S L 3/2. Then
G 6
An
f(zr. Vr) - - [iv((a
vf(zev))
$ Ia
Sfo ft [ n²
Ly
+ nh maxx] + a(w) T.
where o(1) < St Tegd. small error term when p is chosen small.
17

---

Published as a conference paper at ICLR 2026
Proof. For arbitrary stopping time т, start from the smoothness given by Lemma 1:
((1+41,8++)) - f((2,, M1)
V (f(e,, Mel,
xe+1 Zi
y1++ yt ) + 2
re+1 - -2 Mt41 - ye!
0
Je+1 Itz
0
Ty
yt+1 yit
Le
Ly
(Val(ar Me), Tt-1 - xx1) + vV (at: vc), Je+1 -xi) +
2
Te+1
-
xr 1l
+
2
|| - '/ell
@ 7and V (1t, yc).
9t yの nn v.. (ate ve); 3) + T1
I n²
St
gt
112
+ 12 I. 21 11 1211
n
n
n
n
he "1 my y h4
(V of ve).
n 11
where (i) we applies the derivation of (38) from Mishchenko et al. (2020) with setting The S 26, - We note that the parameter update doesn't 't involve the gradient estimation; so, keep the original stochastic gradient he for this step. Let & (V f (Z Ut), )) and th 11 representing the errors caused by the zeroth-order gradient estimation.
and f/y
2L,
E2 = b. d=
Then we obtain
(04+1,8(11) f(x,, vi) 7. V v1). 9t
+ 17 L, 21 St
1|
+ &2 + &
n
n
+ 17 Lyn m "
h
hu 112
nn Vn (I Uc),
n
Then we set 1/ 5 2L,
and f7 S 2h,"" By (39) from Mishchenko et al. (2020),
((1441 f( 36)
+ n
VV [
n
2
n 9t Val(se mi)
n
hr
V ye)
+ + Ez.
n
Then we take expectation both sides and decompose (aI( # ||? using Lemma 1 with the Lipschitz constant Lggmax and Vy Il with the Lipschitz constant Ly myy.aaxi more explicitly, we have
V Vas(sa, St
V
Σ Va Inl (esii,
i=1
V (01,0, Ne,00 v &f (1.
1=1
T
1 - Te,e max
Applying
Assumption 2 and Lemma 5 from Mishchenko et al. (2020) to bound
E
E | x 2 we obtain
a
[11411 (1141,1t+1) - [x 11)) V 34 12 nl
nn?
Vyl((r (ate
L2
ny L2
nun v.max
2011 + n 0
y.max
[1 VV n0²=00 20 +EE + E&
2
We re- we-irite this inequality into the matrix form.
Dan 2
f( - -[ v. ) + (V (F))
V ((()))))
0² 2 T1 2.max + y. EE + E&g + V [
I
0 2 v ?
V Vi( v.).
Id
18

---

Published as a conference paper at ICLR 2026
When choosing 7)
and fl 21
I 2oxy
it ensures that
Y/ Jd 0
22 Idz
1. n
L.muox
L.
0
My Id
1
2
0
Id
2
0
n
L2
max
1b
2
Therefore, let A2
be PSD matrix. Then we obtain
0
Myddy
f(1+11, Dc+1) f ve) | 2 o² n² I mm + 1 E + E&
2
Then we apply Lemma bound Eε, and EEg, respectively. By the stopping time construction, we have V=](( w=li I [V ut)ll S G. Therefore, we have
E n V f 31), ge
S 1l2 L (d + 3)3/2
Similarly, we have
St 11 E
E 1/2 Ly 12
n
n
32dd $ L
af f 8ell + 108p2
We further simply the inequality by letting 1he 351lznd
Then have
f(E+-1) f(x, Me) [V ve)]
vf(
0
a²
2 n [n
L
y.max + 1/ L2 m.ml + o(w):
where 0(4) represents small error term when tends to O. Lastly, we sum over t < and obtain
f(rr, -(1r.)))- - Σ ((f
1z la
Vf(ze v.)
Engdd
ffu fr +
' n² L
I2
max + nn 22
m + )(w)) T,
which completes the proof. Here, (N s dn by letting μ S L. 22/2-
D PROOF OF THEOREM
Here, we re-state our main theorem with full details.
Theorem 2. Suppose that Assumption I and Assumption hold for the objective function f(x, y) - f(x, ys i) and satisfy the hybrid generalized smoothness properties. Let ó (0, 1) and (()),)))<= be the SGD with Random Shuffling dynamic generated by Algorithm I for solving the optimization problem Eq. (1). Given as
F P=
where fo = fxo, yo) is the initial function value and o' is constant-level value given by Eq. (4) and as
200G2 de
G2 + e
H 2
2
8
19

---

Published as a conference paper at ICLR 2026
define the stopping time
A Tu where T1 min |
f(z++1. Vo+1) f F} T and
T2 == mi ||l||| > H} A T. where & is defined Lemma 6. If learning rates Ti. By and the perturbation stepsize μ are chosen such that
2 2 Zrr.mmnen 384L,ndd V
n S min
Tonliz.max
1
2
Fy I min
V onlyy.aax
2Iy
G 6
1
S min
L d3/2 3LgIndG
where all constant G, maxx Ly maxs Los Ly are defined relying on F with presented in Lemma 4, and the maximum number of epoch is chosen as
G2
fo f +
T2
+
8
n
then with the probability at least 1
8,
Σ
E
(1,,111)
T
k<
Proof. Let A ( Eier VV (ze, 31 w0 S se and B = fr T be two events. We consider
the following lower bound of the probability of event A by conditioning it on the event B:
IP(A) Z P)AnB) = ()A
> [1 P B)] IFBB))
Our goal show that the probability of Ex<r IVV (ot, ye )=l 2 2 T)
T (the event A
and (r T} (the event B3) both small. We bound each term separately.
First, we bound the probability of {+ Eter IVV (2r, ve)
By Lemma 6,
we let
a² n [
T. m. - L2 mmax + o(1)) T.
(4)
2
If the event is conditioned on 2 T, we always have [V/ S Gor = 1,,,...,,
1, where G determined by Lemma Then we obtain
P
VV (0,(
P
(4)
[
- T2 /ec
E
(0))
G²
Σ E
(Ix) )
/05
exp
UKT
G² -c-)
exp
lfo 1fo +
8 -
where (i) takes exponential on both sides, (ii) applies the Markov inequality, (ii) applies
the Hoeffding's lemma, (iv) applies Lemma 6 with setting Tmin min (ro. 3o).
Ri Ny and fo ==
4 3
Before we evaluate the necessary T. we need to choose hyper-paramedt parameters to make o' less than some constant independent d, n, or other crucial constants. To do so, we set
2
2
nu V onnn.max
Tl S V onnyymaxx
1 3L,Tnd,G'
20

---

Published as a conference paper at ICLR 2026
Then we obtain o' s 2 + F/y Let Te² and pminnnto 15o 1---5----- +2 nzlt G²2 it solves
e c
$. Then
G
1
èT 2 In
0
+
8
+
Mmin
[fo - -f- + 21700 + 17yl
2
G
[o f 2n + 7l
26-22
22
8
Muiin n2
T). Recap that we consider the stopping time defined as T = Ti A 72, where TI == mine (t ((141139-1)) - > F} A T and
Then we bound the probability P(B²) = P(T <
72 min {t) |&ll| > H3 AT. Here, E, is defined as
c Vi
V( - (5)
est. etr.
stoc. en.
We note that for the last dy entries, the estimation error term since we do not apply gradient estimation for this part. Both F and H in the definition of stopping times will be determined later. Then we notice that
PP B²) = P(T < 7) = P((n < { <T)
P( <) P(Ti <Tv T).
We bound each term separately as follows:
Choose such that PP(T2 < 7) S f: We have
P(T2 <) P
U [1Gll H}
WTT
P (llee)) H)
3 E
Elgg: 9.112 + +3E1 2 V
(If v:)l
3E 4x
Vy Vyf ue)
122
(1)
3
[64d Va.f vi]1 + 216 L2 ma.a d HH2
(3L2 maxx
1ee + 3L )
[n + no*1 /12 T
(184) 200 de + 2G
112
where (i) applies the Markov inequality, (ii) applies Lemma and Lemma 5 from &G
Mishchenko et al. (2020), and (iii) we choose sufficiently small μ S L, m. and
learning rates ne S V3L.mman" and B7
simplify the upper bound.
V3Ly.mme
1200G de +20 + T
Then we choose
6 It solves
H²
1200g2d de
H = 21
(6)
o Choose F such that P( < T, 2 T)
< 4.
Because {n < T, 2 T} c
(f(I.. V+) f P)
P(T < T, T2 > < ((f(z Ur) - f F
7
@2 2E (5+, vr) - 1
2 f + d'//F.
21

---

Published as a conference paper at ICLR 2026
where (i) applies the Markov inequality. Let 6 4 2[f(2o) f + o'//F. It solves
F F= 3 (f((z) - + o
(7)
Combining both upper bounds with choosing H and defined by Eq. (6) and Eq. (7), respectively, we have
P(B)) (P(T < T) - 8
2
Then we obtain the lower bound of P(An B) follows:
P(An B) = P(A B) 2 [1 P[A1BB[[1P(BB P(B))
2 [1 2 [[ -1
2 21-6
Lastly, discuss the hyper-parameter choices and the epoch complexity. To make Lemma 6 hold, -
we have set The I min 2lp. 384L_nd, } and fly I 2by.maen and the perturbation stepsize
S T E7 When bounding the probability of { >.eT IIV (me, xt)1 > e² 7 T}
and the
probability of P(r < T), we additionally require
2 T onter,aax
Ta S minn
V/L maxn
2 V
72 S min
T onLyymax
V3Iry
&G
/ S mind
3LnTndoGG
Lu,
3/2
Therefore, in summary, we have
Ta I min
2 384L
T onLez,max
1
Fy S min
m V T onLy.max
G
1
# S min
L de
IL,Tnd,
Under these hyper-parameter choices, we also need to require
G
fo f 2n
nl
T2
8
Maiin n
where
Mmii
min ).
ensure that the probability of small (less than
2 We observe that by simply setting Mmin 2 (we always make by choosing T O n² )), the above condition T degenerates to T 2 0 O(=²)) Therefore, it concludes that if T (c/-///4))))) 4//n. with the probability at least 1 5,
Eter /IV/ (Ie, 11
2 r 2 7
1 T RT
(Z:)) 5e
Then the proof is completed.
Here, we discuss how we determine the optimal value Timin O(e2). general, we can set Mnin
O(E"), which leads to the condition on T:: T' 2 O(E ((f=-2 A smaller is always better. However,
we need to ensure the learning rate condition is satisfied; that is, Mmin I <0 F). It solves T < We let - ~2x 2 -2-a which solves 2 2. Therefore, when 7win O(e²), the complexity is optimal and attainable.
22

---

Published as a conference paper at ICLR 2026
SST2
Copa
WinoGrande
steps
steps
steps
ZO-FT
1.1 x 10*
10-5
1.6 x 10*
10
1.8 x 10*
10
FO-Prompt
6 10a
x 10a
10a
Hlybrid-Prompt
1.5 10
10-3
x 10
10-5
9 x 10
10-3
Llama-2-7b
FO-Prefix
2 x 10a
1.5 x 10a
3 x 10y
Hyhrid-Prefix
9.5 10a
10
7.5 x 103
10"
9 x 10a
10
FO-Lora
2 x 10*
2.5 x 103
2.5 x 10a
Hybrid-Lona
1.6 10a
10-3
1.15 x 10*
10-5
4.5 10
10-5
70-FT
1.0 10*
10
x 10a
10
1.75 x 10*
10
FO-Prompt
2 x 10*
1.3 x 10*
2 x 104
Hybrid-Pronne
2 x 10a
10-5
1.5 x 10
10-5
2 x 104
10-6
Vicuna-7b-v1.5
FO-Prefix
2 103
X 104
2 x 10a
Hybrid-Prefix
2 x 10
10-5
1.7 x 10*
10-5
4 x 10
10-5
FO-Lora
2 x 103
x 103
3.5 10
Hybrid-Lora
2 104
10-5
10
10-5
10
10-5
ZO-FT
2 x 10*
10-5
8.5 10a
10-4
10
10-5
FO-Prompt
2 10*
1.6 x 10*
9.5 10
Hybrid-Prompt
2 104
10-5
x 10
10
1.4 104
10-5
OPT-1.3b
FO-Lora
3 10a
1.9 104
1.45 10*
Hybrid-Lone
4 x 103
10-5
1.9 x 10a
10-5
3 x 10a
x 10-4
FO-Prefix
2 104
x 10a
9.5 103
Hyhrid-Prefix
8.5 x 103
10
1.15 x 10*
10
2 x 104
10
Table 5: A detailed breakdown of the optimal hyperparameters including training steps and μ spec- ified in Eq. (2) and training specifics for each fine-tuning method applied to different model archi- tectures across SST2, Copa, and WinoGrande tasks. Highlighted cells indicate efficient training processes, showcasing the reduced steps required by hybrid approaches to achieve optimal perfor- mance.
E EXPERIMENTAL DETAILS
In this paper, we evaluate our proposed hybrid-tummi -tning method across a diverse spectrum scenarios including three distinct tasks, three transformer-based language models, and three PEFT methods. This extensive exploration not only demonstrates the broad applicability of our approach but also provides robust evidence for its effectiveness and versatility in enhancing model performance across various domains and architectures. In this section, we will briefly review these components and delve into more details of our experiment settings.
E. OVERVIEW OF TASKS
In this section, we briefly discuss the task we consider in our paper. All of tasks are ready to use in the ZO-Bench code base Zhang et al. (2024) and we follow the default setting and the same train/tes/valiiationn split of their original implementations.
Text Binary Classification In this paper, we consider the Stanford Sentiment Treebank v2 (SST2) dataset Socher et al. (2013) and the Word-In-Contex (WIC) dataset Pilehvar Camacho-Colladd (2018), which presents the simplest binary text classification problem. The SST2 dataset is suffi- ciently simple and convenience to use to verify our motivating examples (as demonstrated in Fig- ure la and Figure lb). The WIC dataset provides more challenging task that requires understanding word meanings in different contexts. Both datasets serve as excellent benchmarks for evaluating the performance of our proposed methods in binary text classification tasks.
23

---

Published as a conference paper at ICLR 2026
Question Answering The Choice Of Plausible Alternativss (COPA) dataset Roemmele et al. (2011) is a common benchmark for evaluating the commonsense causal reasoning ability of a lan- guage model. It contains one thousand English-language questions answer pairs. We choose this task to evaluate our approaches in improving the question-answering capabilities of models, partic- ularly in scenarios requiring causal inference and commonsense reasoning.
Common Sense Reasoning Task We consider the WinoGrande dataset Sakaguchi et al. (2021) and the Winograd Schema Challenge (WSC) dataset Levesque et al. (2012), which present a chal- lenging common sense reasoning task. The WSC dataset is designed to evaluate machine under- standing and reasoning capabilities by presenting pronoun disambiguation problems that require human-like inference. The WinoGrande dataset is designed be more difficult and larger-scale version of the original Winograd Schema Challenge, requiring models to demonstrate human-like like reasoning capabilities. By including WSC and WinoGrande our experiments, aim assess how well our approaches can enhance a model's 's appropriate inferences based on contextual information.
ability to reason about complex scenarios and make
E.2 OVERVIEW OF PEFT MODULES
In this paper, we mainly consider three types of PEFT modules. In our proposed hybrid-tuning ap- proach, we jointly train one of these PEFT modules with the base LLM to improve the convergence and overall performance. The following paragraphs provide detailed overview of the three main PEFT modules considered in this study: Prompt Tuning, Prefix Tuning, and Low-Rank Adaptation (LoRA). our experiments, we follow the default configuration of Zo-Bench code base Zhang et al. (2024) without making additional modifications. It is worth noting that our hybrid-tuning methods are also applicable to other recently developed PEFT techniques including (1) other LoRA vari- ants such as X-LoRA Buehler & Buehler (2024). Zhang et al. (2023a), LoHa Hyeon- Woo et al. (2021), and LoKr Yeh et al. (2023); (2) other soft prompts techniques such P-tuning Liu et al. (2021; 2023); and (3) Infused Adapter by Inhibiting and Amplifying Inner Activation (IA3) methods Liu et al. (2022).
Zhang et al. (2023b), AdaLoRA
Prompt Tuning Prompt tuning Lester et al. (2021) is lightweight fine-tuning method that prepends trainable continuous prompt tokens to the input. These prompt tokens are optimized during training while keeping the pre-trained language model parameters frozen. This approach allows for task-specific adaptation with a small number of parameters. Prompt tuning is particularly effective for large language models and can be seen as a form of soft prompting that learns optimal input representations for specific tasks.
Prefix Tuning Prefix tuning Li & Liang (2021) extends the concept of prompt tuning by adding trainable prefix tokens not only to the input but to each layer of the transformer model. This method prepends a trainable continuous prefix to the keys and values of the self-attention layers in each transformer block. By doing so, prefix tuning allows for more flcxible and expressive task-specific adaptations compared to prompt tuning, while still maintaining relatively small number of trainable parameters.
LoRA Low-Rank Adaptation (LoRA) Hu al. (2021) a parameter-effiiiet fine-tuning method that adds low-rank decomposition matrices to the weights of the pre-trained model. Instead of di- rectly updating the model's weight matrices, LoRA introduces pairs of rank decomposition matrices for each weight matrix being tuned. These low-rank matrices are initialized randomly and trained to adapt the model to specific tasks. LoRA significantly reduces the number of trainable param- eters while maintaining competitive performance compared to full tuning.. It offers several advantages, including faster training, lower memory requirements, and the ability to switch between multiple fine-tuned tasks efficiently by changing only the LoRA parameters.
E.3 CONVERGENCE OF HYBRID FINE-TUNING
In this subsection, we present the training curves (including the training loss, validation accuracy, and the test accuracy) for 3.T-1.3B Zhang et al. (2022) model on SST-2 Socher et al. (2013) dataset in Figure 4. We observe that significant efficiency gain in terms of training steps. The hybrid
24

---

Published as a conference paper at ICLR 2026
Train
Validation Accuracy
Test Accuracy
Hybrid Tuning
0.8
0.90
0.90
FO-Prompt
0.85
0.85
Train Laxs
0.6
Accuracy
0.80
Accuracy
0.80
0.75
0.75
0.70
0.70
0.2
Hybrid Tuning
0.65
Hybrid Tuning FO-Prompt
0.65
FO-Prompt
0.60
Z0-FT
Z0-FT
0.0
0.60
0.55
10000 15000 Steps
20000
5000
10000
15000
20000
5000
15000
20000
Steps
Steps
(a) Training curves OPT-1.3B model with the prompt tuning on the SST2 dataset with using the optimal hyper- parameter indicated in Table 5. The hybrid-tuning achieves the significant better performance. Notably, this phenomenon is also observed in other tasks and for other model architectures.
Validation Accuracy
Test Accuracy
0.69
Hybrid Tuning
Hybrid Tuning
2.6
ZO-FT
0.76
FO P0-Prompt
FO-Prompt
2.4
0.74
0.67
Tainn Loss
Accuracy
0.66
2.2
2.0
0.68
0.64
0.66
Hybrid Tuning
FO-Prompt
1.6
ZO-FT
0.62
5000
10000
15000
20000
5000
10000 Steps
15000
20000
5000
10000
15000
20000
Steps
(b) Training curves for Vicuma-7b-v1.55 model with the prompt tuning on the WinoGrande dataset.
Figure 6: Comparison of training curves for different models and datasets. These results demonstrate that the similar outperformance of hybrid-tuning observed across various model architectures and NLP tasks.
method consistently achieves optimal performance regarding training loss. This trend is observed across different tasks, PEFT methods, and model architectures, suggesting that the efficiency of hybrid tuning scales well (e.g. for Reuna-7b-v1.5 model on the WinoGrande dataset in Figure 6b). A detailed breakdown of is provided in Table 5.
E.4 ESTIMATING SMOOTHNESS
In Figure la and Figure 1b, the smoothness of the loss landscape of the OPT-125M (and the LoRA module) is estimated by approximating the norm of Hessian matrix at the stochastic data point using the zeroth-order gradient estimation to the Hessian-vector products (HVPs):
Vf(x hv; (x:E))
Hessian( (z)
CEBatch
where fz(z; {) is the stochastic gradient at for the data point & in the given data batch, / is a small perturbation size, and random unit vector. We estimate the Frobenius norm ||Hessianz)) II VEu H2 of the Hessian by sampling multiple random vectors and computing these HVPs.
For Figure la, initialize the parameter of pre-trained binary classification OPT-125M model and train it over the SST2 dataset for 5000 steps with setting the learning rate r7 = 5 x 10 and the batch size 8. We sample 100 independent vectors from the unit sphere estimate the HVP with the perturbation l = 1 and obtain the Hessian norm as the approximation of the local smoothness constant L.
For Figure 1b, we initialize the parameter of pre-trained binary classification OPT-125M model as the base model and randomly initialize the LoRA module with the rank 16 and the LoRA Alpha 32
25

---

Published as a conference paper at ICLR 2026
(the detailed configuration be found in the source code) and jointly train both components over the SST2 dataset for 5000 steps with setting the learning rate 7) = 5 x 10 and the batch size 8. We collect all parameters along the SGD trajectories. We perturb the parameter of the base LLM and the LoRA module, respectively, with 100 independent vectors from the unit sphere and the perturbation h 10-5 estimate the smoothness.
E.5 OMITTED EXPERIMENTAL SETTINGS
Following the methodology of Malladi al. (2023); Zhang et al. (2024), we assessed our approach on 6 representative NI.P tasks including the sentiment classification task on the SST2 dataset Socher et al. (2013), the sentence differing task on the WSC dataset Levesque et al. (2012), contextualized word and sense representation and word sense disambiguation task on the WiC dataset Pilehvar & Camacho-Collados (2018), the question answering task the COPA dataset Roemmele et al. (2011), I), and the common sense reasoning task on the WinoGrande dataset Sakaguchi et al. (2021). The models we use in our experiments include OPT-1.3b Zhang al. (2022), Vicuna-7b Chiang et al. (2023), and LLaMA-7b Zhang et al. (2023b). We compare the performance of our approach against standard PEFT methods including first-order prompt tuning Lester et al. (2021). LoRA tun- ing Hu et al. (2021), and prefix tuning Li & Liang (2021). For cach dataset, we randomly sample 1,000 examples for training, 1,000 examples for evaluation, and 100 examples for development. Performance is evaluated using accuracy or F1 score, appropriate for each task. All experiments utilize SGD as the optimizer. In the case of prompt tuning and prefix tuning, the prompts are ini- tialized according to the predefined settings in Table E.2 of Malladi et al. (2023), while for LoRA tuning, we initialize with zeros. We perform hyperparameter tuning for all methods and report the best configurations. For all methods, set the maximum number of training steps to 20,000, with early stopping applied when applicable. The detailed hyperparameter setting, overviews of the tasks and PEFT methods, syper-parameter setting, and the full results are reported in the supplementary materials.
For the zeroth- order approximation, follow the same approach outlined by Malladi et al. (2023). In the case prompt tuning and prefix tuning, the prompts are initialized according to the pre- defined settings in Table E.2 of Malladi al. (2023), while for LoRA tuning. we initialize with zeros. We perform hyperparameter tuning for all methods and report the best configurations. For all methods, we set the maximum number of training steps 20,000, with early stopping applied when applicable.
E.6
SEARCHING
In our experiments, we conducted systematic grid searches across all combinations of tasks, models, and PEFT methods. For FO PEFT training configurations, we primarily rearche the learning rate (among 0.001, 0.0001, 0.00001, and 0.000001), while maintaining fixed hyperparameters for LoRA (rank=8, alpha=16) and Prompt Tuning (10 virtual tokens). In hybrid training configurations, we adjust the search space to include both the base learning rate (0.001 and 0.0001) and the zero- order (ZO) learning rate (10-* and 10-7), creating a two-dimensional grid search with the same number of hyper-parameter as the FO method. Notably, our hyperparameter configurations are inspired by theoretical analysis of hybrid -uning (i.e. the base LLM requires a smaller learning rate), which allows us to strategically constrain the grid-search space. This theoretical guidance not only reduces computational overhead but also demonstrates how theoretical insights can effectively streamline the practical implementation of fine-tuning procedures.
All experiments maintained consistent training parameters including 5 epochs, batch size of 16, and 20,000 maximum steps, with evaluation performed every 500 steps. The search strategy was implemented using grid search methodology, with accuracy on the validation set as the optimization metric.
F EXTENDED DISCUSSIONS ON WALL-CLOCK TIME
While standard zero-order methods often accelerate training by bypassing backpropagation (Malladi et al., 2023; Zhang et al., 2024), our Hybrid Tuning maintains comparable efficiency by restricting the expensive first-order updates solely the PEFT module. This design ensures minimal com-
26

---

Published as a conference paper at ICLR 2026
Smonthed Thain
Validation Seconds
Test Seconds
0.9
Hyorid Tuning
0.8
0.90
0.90
0.7
20-FT
0.85
0.85
0.6
0.80
0.5
0.75
Train
0.75
0.4
0.3
0.70
0.65
Hybrid Tuning
Hybrid Tuning Fo Prompt
0.2
0.65
FO Prompt
0.50
0.1
0.60
0.55
5000
20000
15000
20000
10000
15000
20000
10000
15000
20000
Seconds
Seconds
Figure 7: Training performance of OPT-1.3B on SST-2 using prompt tuning versus wall-clock time. The time cost of each method is: Hybrid Tuning (1.28 sec/step), FO-Prompt (0.80 sec/step), ZO-IT (0.64 sec/step), and one forward pass (0.15 sec/step).
putational overhead despite requiring both forward-pas estimation and backpropagation. Empirical tests Llama-2-7b confirm that our method (2.36 sec/step) remains competitive with full-parameter FO-SGD (2.27 sec/step) (one forward pass costs 0.34 sec/step). To visualize this efficiency, Figure plots the training curve of OPT-1.3B (with prompt tuning) on the SST-2 dataset against wall-clock time, as the comparison against Figure 4.
G THEORETICAL CONTRIBUTIONS
On the thcoretical side, our work addresses key gaps in the current optimization literature.
First, introduce and analyze SGD) under novel hybrid smoothness condition (Defi nition 1), which generalizes both classical I-smoothness assumptions and (-generalized smoothness assumption; this condition better reflects the heterogeneous structure of mod- ern hybrid models. To our best knowledge, this is the first formal treatment of SGD under such condition.
Second, we extend the analysis the random reshuffting setting, marking the first conver- gence result that integrates generalized smoothness with reshuffling-aard based SGD algorithms.
Finally, we improve the known sample complexity bounds for SGD under generalized smoothness by applying sharper concentration techniques. This leads to a provable im- provement in the dependence on the confidence parameter 8. reducing from O(E)//8) to OlE 2/8 +
H BROADER IMPACTS
The proposed hybrid fine-tuning framework has the potential to broadly impact the development and deployment of LLMs by addressing both computational efficiency and adaptability to new tasks. By combining ZO optimization for the base LLM with FO optimization for PEFT module, the approach enables scalable and memory-efficient training without sacrificing performance. This hybrid strat- egy introduces novel theoretical framework-tte the hybrid smoothness condition that rigorously accounts for heterogeneous parameter landscapes, offering insights relevant not only to NLP but also general large-scale machine learning systems. The framework could facilitate broader acces- sibility of LLM fine-tuning in resource-constrained environments and inspire future work on hybrid optimization in other domains.
I
LIMITATIONS
While our hybrid fine-tuning approach demonstrates strong empirical performance and theoretical convergence guarantees, it has several limitations. First, the effectiveness of the method relies
27

---

Published as a conference paper at ICLR 2026
tuning separate learning rates for the base LLM and PEFT modules, which may require additional hyperparameter search. Second, the ZO optimization used for updating the base model, though more memory-effieiane can still be computationally expensive due to repeated function evaluations, especially for large-scale models. Finally, the current formulation is limited to joint training of LLMs with PEFT modules and may not generalize directly other forms of model composition, such as mixture-of-agent or other multi-agent systems. Addressing these challenges remains an avenue for future research.
J LLM USAGE
We primarily employed large language model (LLM) an auxiliary tool to support the prepara- tion of this manuscript. The LLM was used for tasks such as language polishing, improving clarity, and suggesting alternative phrasings. did not generate new research ideas, perform data analy- sis, or contribute substantively to the scientific content of the work. All conceptual development, methodology, experimental design, and interpretation of results were carried out independently by the authors. The authors take full responsibility for the content of this paper, including all text revised with the aid of the LLM. The LLM not considered an author or contributor.
K CONCLUSION
conclusion, this work introduces a novel hybrid fine-tuning approach for LLMs that combines zeroth-order optimization for the base model with first-order optimization for PEFT modules. Mo- tivated by the hybrid smoothness condition of our hybrid fine-tuning system (Definition 1), we develop theoretical framework centered on this theoretical challenge introduced by the hybrid fine-tuning method. Our empirical examples (Section 2.2) and convergence analysis (Theorem 1) demonstrate the necessity of applying different learning rates for different modules. Our analysis achieves the best-known sample complexity under much milder conditions in the existing litera- ture. Extensive empirical evaluations across multiple NLP tasks, model architectures, and PEFT techniques validate the theoretical insights and show consistent performance gains over traditional fine-tuning methods shown Table By addressing fundamental challenges in joint LLM and PEFT training, our work opens new avenues for efficient LLM fine-tuning and provides a solid foundation for future research on optimizing hybrid systems in machine learning.
28


Reviews:
[{"review_id": "aRxyMdU3iF", "reviewer": "Reviewer_ASAc", "summary": "The paper proposes \"hybrid fine-tuning,\" a novel method that jointly updates a base LLM ($x$) and a PEFT module ($y$). The core idea is to use a zeroth-order (ZO) optimizer for the large base model ($x$) and a standard first-order (FO) optimizer for the small PEFT module ($y$). The authors motivate this with the \"hybrid smoothness condition\" and provide a convergence analysis, along with strong empirical results showing performance gains over FO-PEFT methods without additional memory overhead.", "questions": "Q1:  Regarding W1: Could the authors provide some insight into the wall-clock time comparison? Given the $\\sim$2x FLOPs per step, how does the impressive reduction in steps (e.g., in Fig. 4) translate to actual training time savings?\n\nQ2: Regarding W2: Could the authors comment on the gap between the theoretical bounds (e.g., $\\eta_x \\propto 1/d_x$) and the practical learning rates used (e.g., $\\eta_x = 10^{-6}$)? Does this suggest the practical loss landscape has a structure (e.g., sparsity, low-rank) that the theory does not yet exploit?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Novel Method with Excellent Motivation: The core idea of combining ZO (for the base model) and FO (for the PEFT module) is a novel and highly insightful solution. The paper provides an exceptionally clear and compelling motivation, empirically demonstrating the vastly different smoothness landscapes that necessitate this hybrid, multi-learning-rate approach.\n\n2. Significant Practical Value and Strong Empirical Gains: A key achievement is that this method provides superior performance without incurring any additional GPU memory overhead (Table 2). This practical value is backed by strong and remarkably consistent experimental results.\n\n3. Solid Theoretical Contribution: The introduction of the \"hybrid smoothness condition\" (Definition 1) is a solid contribution to the optimization literature. It provides a formal language to analyze this new class of heterogeneous optimization problems, even if the resulting bounds are not yet perfectly tight.", "weaknesses": "1.  Clarification Needed on Wall-Clock Time: The paper's efficiency claims are focused on \"steps to converge\" (Fig. 4). However, the ZO estimator (Eq. 2) requires two forward passes, implying a $\\sim$2x computational cost per step. A clarification on the real-world wall-clock time trade-off would strengthen the paper's practical claim.\n\n2. Gap Between Theoretical Bounds and Practice: There appears to be a gap between the derived theoretical bounds and the practical implementation. The proof (Theorem 2) suggests a dependency on model dimension $d_x$ (e.g., $\\eta_x \\propto 1/d_x$), which would lead to an extremely small learning rate. However, the experiments successfully use a much larger constant $\\eta_x = 10^{-6}$. This suggests the theory, while proving convergence, does not yet fully capture the practical power of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eU8knxhHnq", "reviewer": "Reviewer_6rr7", "summary": "This paper proposes a novel hybrid fine-tuning paradigm for LLMs that combines zeroth-order optimization (ZO) for the base LLM parameters with first-order optimization (FO) for PEFT modules. The authors introduce a new theoretical framework called the \"hybrid smoothness condition,\" which accounts for the heterogeneous optimization landscape of joint LLM and PEFT training. The paper provides convergence analysis for SGD under this hybrid condition and conducts extensive experiments across six NLP tasks and three LLM architectures. The proposed method demonstrates improved efficiency and competitive performance compared to traditional fine-tuning methods while maintaining memory efficiency.", "questions": "Could authors provide some more experiments on LoRA + Adam (performance and memory)?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The hybrid smoothness condition and the convergence analysis for SGD under this condition are significant advancements. The authors also provide sharper complexity bounds compared to prior work.\n\nThe paper evaluates the proposed method across six NLP tasks, three LLM architectures, and multiple PEFT techniques, demonstrating its broad applicability.", "weaknesses": "Hybrid LoRA does not show significant improvements over standard LoRA in many tasks. This raises concerns about whether the added complexity of hybrid fine-tuning is justified.\n\nThe paper does not compare memory usage and performance with LoRA + Adam, which could provide a more nuanced understanding of the trade-offs between performance and efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ipP2Kk30Bk", "reviewer": "Reviewer_o27F", "summary": "This paper studies hybrid fine-tuning for large language models where two sets of parameters are trained together: 1) The base LLM parameters x, and a small PEFT module y. The authors observe that using the same learning rate for both is usually problematic: if the learning rate is tuned for the base model, the PEFT module learns too slowly but if it is tuned for the PEFT module, the base model becomes unstable and diverges. To explain this, they introduce a hybrid smoothness condition, which says that the loss landscape behaves very differently along the x and y directions. Using this assumption, they prove that convergence requires different learning rates for the two parameter blocks. They give upper bounds for each learning rate and show that the best practice in real LLM fine-tuning (small LR for x, larger LR for y) has theoretical justification. They conduct experiments to support the theory using prompt tuning and full fine-tuning baselines.", "questions": "1, How confident are you in the hybrid smoothness assumption? Can you provide empirical evidence showing that geometry for x is worse than for y across multiple datasets and models?\n2, What happens if the PEFT dimension grows? Does the theory still guarantee stability when dy​ becomes large (e.g., high-rank LoRA, prefix tuning, or side-tuning)? If not, how should the learning-rate bound be modified?\n3, Can you validate on more realistic settings? Would you extend the results to common RLHF/SFT datasets, and at least one (more recently released publicly available) model larger than OPT-1.3b?\n4, Can you give a recommended ratio between learning rates of x and y, or a heuristic based on model size? How should practitioners choose the two learning rates in practice?\n5, How does the theory interact with optimizers? Would adaptive optimizers like AdamW weaken or strengthen the LR asymmetry?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1, The paper explains a real and common painpoint in LLM training: why PEFT often requires a higher learning rate than full fine-tuning. The theory directly explains the practical behavior seen in experiments.\n2, Their work reveals that “use different learning rates for base and PEFT parameters”, which is highly practical and easy for practitioners to apply.\n3, The loss curves clearly show the need for asymmetric learning rates and support the theory.", "weaknesses": "1, The hybrid smoothness assumption is new, but the paper doesn’t fully explain where it is justified. Moreover, applying a similar argument as in the paper, one can even claim that different modules of the backbone model favors different learning rates. The paper however doesn’t touch on this point.\n2, All experiments use prompt encoder tuning on a sentiment dataset. It is not clear whether the theory holds on other tasks or other PEFT methods.\n3, The learning rate bound for y in Thm. 1 doesn’t depend on d_y, while the bound for x depends on d_x. It’s unclear from the statement why this is the case. After all, as d_y becomes larger this likely will stop to hold at some point.\n4, Many existing recipes already use different learning rates for different parameters groups. So the paper should clarify what is new about its methodology.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)