Towards Dynamic Interleaving Optimizers
Authors: Yile Chen
Zeyi Wen
Jian Chen
Jin Huang
Venue: ICLR 2026 Poster (2026)
Keywords: HPO
optimizer
Primary area: optimization

Abstract: Optimizers are critical for training deep neural networks. Existing training processes rely on a single static optimizer (e.g., SGD) or a simple hybrid of two optimizers, which miss the opportunity to exploit evolving dynamics in different training states, degrading model quality and convergence. In this paper, we propose a novel dynamic optimizer switching method called **D**ynamic **O**ptimizer **I**nterleaving **T**raining (DOIT) method, which builds surrogate models to predict different optimizers' performance from current parameter states. DOIT uses an acquisition function that combines the results from surrogate models with transferability assessments and process information to select a suitable optimizer for the subsequent training. Experiments on various models and tasks (e.g., image and text classification, machine translation, and object detection) show that DOIT effectively enhances the training, achieving faster convergence (i.e., 2\% to 10\% faster) and higher accuracy (i.e., 1\% to 3\% improvement). Additional independent experiments and case studies further validate DOIT's effectiveness.

Published as a conference paper at ICLR 2026
TOWARDS DYNAMIC INTERLEAVING OPTIMIZERS
Vile Chen Zeyi Wen Jian Chen'; JJin Huang'
South China University of Technology, 2HKUST (Guangzhou), *HKUST,
"South China Normal University jireh. gmail com, wenzeyi 1eheestttaeeeeeeeeee -erestt-gzeeee edu. en, el ellachen@scut edu.. cn, huangj i nêm. scnu. edu. cn
ABSTRACT
Optimizers are critical for training deep neural networks. Existing training pro- cesses rely on single static optimizer (e.g.. SGD) simple hybrid of two optimizers, which miss the opportunity to exploit evolving dynamics in differ- ent training states, degrading model quality and convergence. In this paper, we propose a novel dynamic optimizer switching method called Dynamic Optimizer Interleaving Training (DOIT) method, which builds surrogate models predict different optimizers' performance from current parameter states. DOIT uses an acquisition function that combines the results from surrogate models with trans- ferability assessments and process information to select a suitable optimizer for the subsequent training. Experiments on various models and tasks (e.g.. image and text classification, machine translation, and object detection) show that DOIT effectively enhances training, achieving faster convergence (i.e., 2% to 10% faster) and higher accuracy (i.e., 1% to 3% improvement). Additional independent experiments and case studies further validate DOIT's effectiveness.
INTRODUCTION
The choice of optimizer and its hyperparameter settings can significantly influence both model qual- ity and convergence in training deep neural networks (DNNs) (Soydaner 2020) Hassan al., 2023). Researchers have proposed various optimizers for better model training (e.g., SGD and Adam and have found that different optimizers exhibit distinct characteristics and are suited to differ- ent tasks. For instance, SGD tends to offer better generalization and often used for head fine- tuning (Poojary & Pail 2019), while Adam converges faster and is commonly used in LoRA (Hu) ctt al., 2021). Moreover, considering the distinct strengths of various optimizers, some researchers have explored combining them enhance training performance (Keskar & Socher 2017).
However, recent studies have found that different op- timizers not only behave differently different tasks, but also exhibit unique characteristics and optimiza- tion strategies in different training states (Im al. 2016). For example. Figure [ presents the optimiza- tion results of three optimizers with varying runs and hyperparameter settings on the Ture Camel function, revealing that: "different optimizers are suitable for different training states (or parameter states)". Static optimizers and simple hybrid methods struggle to exploit the training dynamic, often leading to unstable model quality and increased training costs Sun 2020
SGD
Adagrad
Adam
best
start
(1.0,-09)
(1.6.0.9)
(1.2,0.0)
Figure 1: The different training processes with various optimizers.
To address this problem. we propose Dynamic Optimizer Interleaving Training (DOIT) method, which dynamically selects the suitable optimizer according to the current training state. Through- the training process, DOIT constructs and continually updates surrogate models for different optimizers to estimate their optimization performance under various parameter states. Meanwhile, DOIT calculates score for each optimizer using special acquisition function that combines the
"Corresoonding Authors

---

Published as a conference paper at ICLR 2026
estimated performance, transferability assessment, and training process information. By carefully selecting the optimizer with the highest score during training, DOIT achieves faster convergence and better model quality. To summarize, the key contributions of this paper are as follows.
We investigate the distinct strengths and optimization directions of various optimizers across dif- ferent parameter states. Furthermore. we demonstrate that combining different optimizers during training can help achieve higher-quality models and better convergence.
We propuse Dynamic Optimizer Interleaving Training (DOIT), a method that adaptively switches optimizers based on parameter states during training. DOIT estimates optimizer performance under different parameter states by constructing Gaussian surrogate models and calculates scores using well-designed acquisition function. By iteratively selecting the optimizer with the highest score, DOIT produces higher-quality models with faster convergence.
We implement DOIT and conduct experiments on various models and tasks, including full training and PEFT. Experimental results demonstrate that DOIT effectively improves the training perfor- mance, achieving faster convergence (i.e., 2% to 10% faster) and higher accuracy (i.e., 1% to 3% improvement). Furthermore, present additional case studies and independent experiments to further validate and analyze the performance of DOIT.
RELATED WORKS AND BACKGROUND
In this section, we provide the background of our work, including optimizers, simple hybrid meth- ods, and AutoMI., After that, we identify the limitations of existing approaches.
Optimizers. The optimizers and their hyperparameters are crucial for training DNNs. Tradi- tional gradient descent computes gradients over the full dataset and updates parameters accord- ingly (Ruder 2016). Building on this, various optimizers have been proposed, such as momen- tum (Sutskever et al., 2013) and Nesterov Accelerated Gradient (Qu & Li 2019). Figure iillustrtess illuutrales a portion of the historical development of these optimizers. Beyond these, researchers have also pro- posed various second-order optimizers, such L-BFGS (Liu & Nocedal 1989). K-FAC (Martens) & Grosse 2015). and AdaHessian (Yao et al., 2021). However, due to their practical application challenges, the second-order optimizers are not further discussed in this paper. Additionally, recent work has further introduced learned optimizers via meta-lcarning (Harrison et al. 2022).
Adadelta
NAdam
Padam
AdamP SAM
SGD
Momentum
RMSprop
Adam AdaMax
AdamW SWATS
AdaBound
Adan
NAG
Adagrad
AdaBelief
Adai
1950s
1983
1999
2011
2012
2015
2016
2017
2018
2019
2020
2021
2022
2024
Figure 2: The development of neural network optimizers.
Hybrid optimizers. In addition to in-depth studies individual optimizers, researchers have also proposed simple hybrid methods that integrate different optimizens-typically combining the strong generalization capability of SGD (Sutskever
(2013) with the rapid convergence offered by adap- tive learning rate methods (e.g., Adam (Kingma & Ba, 2014) and Adagrad (Duchi et al., (201]) l)) For example, SWATS (Keskar & Socher (017) switches from Adam SGD for better results. Padam (Chen et al. 2018) blends the two using a partial adaptive parameter, while AdaBound (Luo al. 2019) uses dynamic learning rate bounds for smooth transitions. In addition, AGD Yuue al.,
2023 proposes a switch method between SGD and Adam by the preconditioning matrix. In addi- tion, several methods combine traditional optimizers with learned optimizers to address the inherent stability and out-of-distrbbutee challenges of purely learned optimization, such as HUB (Dai al. 2024) and LGL20 (Prémont-Schwaie et al. 2022).
AutoML and HPO. Beyond optimizers, our work is also closely related to hyperparameter opti- mization (HPO) (Feurer & Hutter 2019) and AutoML (He et al., (021)) Traditional HPO methods static configurations, where a fixed set of hyperparameters is used throughout training, such as random search (Bergstra & Bengio 2012) and Bayesian optimization (Shahriari et al.) (001).. More recent studies have proposed dynamic algorithm configuration (Adriaensen et al. 2022), exploring
2

---

Published as a conference paper at ICLR 2026
areas such genetic algorithms (Biedenkapp et al., 2022) and learning rate control (Daniel al. 2016). Moreover, our work is conceptually related to meta- veeaalea (Vettoruzzo et al. 2024). )). but meta-learning focuses on cross-task adaptation, whereas we focus tuning within single training.
Limitations of current approaches. (i) Static optimizer: While improved optimizers aim to better adapt parameter landscapes, they often incur high computational costs and may not outperform SGD (Keskar & Socher (017). To obtain better models, researchers need to train with different op- timizers, which is time-consuming. Additionally, consistent optimizer throughout the entire training process limits both model quality and convergence speed. (ii) Simple hybrid optimizer: Combining different optimizers can enhance model quality and convergence. However, existing methods merely perform simple switch from one optimizer to another, neglecting the unique strengths of different optimizers under different training states. Such coarse approach not only restricts the stability of the model quality but also impacts convergence speed (Zhuang et al. 2020)
OUR PROPOSED METHOD: DOIT
To better utilize multiple optimizers, we propose Dynamic Optimizer Interleaving Training (DOIT), method that enables dynamic optimizer switching during training. We first provide brief overview of DOIT, followed by a detailed introduction for the key components.
3.1 OVERVIEW OF OUR PROPOSED DOIT
1. ferabiirability weight (Eq.
Dynamic Optimizer Interleaving
Figure illustrates the workflow of DOIT, and a detailed description with pseudocode is presented in Appendix A DOIT calculates the transferability weight wi to assist in the subsequent selection of op- timizers before the training (Step 1). In each opti- mizer switch eycle of iterations (Steps 2-6), DOIT first compresses the model parameters 0f to get the input of the surrogate model (Step 2). Then, DOIT selects the appropriated optimizer of for the current training state (Steps 3-4). Obtaining the training losses, DOIT calculates the performance score and updates the corresponding surrogate model g² (Steps 5-6). By iteratively executing this process, DOIT achieves the dynamic optimizer switching.
2. compress model parameter state
yes
DO
3a. weighted random sample
3b, get acquisition score e (Eq.
select optimizer o*
4. train ilerations get new 8
6n. update sampling weight dl (Eq. 3)
performance score (Eq. 2)
6b. update surrogute models
update respective surrogate model g²
switch cycle
Figure 3: The workflow of DOIT.
For selecting the suitable optimizer (Step 3), DOIT employs methods: weighted random selec- tion and recommendation based on the surrogate model. In the initial training stages, DOIT uses the calculated score to update the sampling weight w,, for randomly selecting optimizers (Steps 3a and 6a). After acquiring enough training results, DOIT selects the optimizers with the highest score e calculated by the acquisition function for each training stage (Step 3b).
3.2 MOTIVATION AND PROBLEM FORMULATION
Before introducing the details of the surrogate model and acquisition function in DOIT, first provide the hypothesis underlying our method: "different optimizers offer distinct optimization di- rections and are suited to different parameter states". Figure illustrates examples of the differ-
ent optimization directions with different optimization functions. It can be observed that although
the optimizers start with similar directions in
the first iteration, their paths diverge after a few iterations. Previous studies have observed this phenomenon, noting that optimizers exhibit dif- ferent optimization directions under varying pa- rameter states (Im al., 2016). Thus, we be- lieve that the optimizer type also requires fine- grained tuning (Adriaensen al. 2022).
SGD SGDM Addggrd
RMSprop Adam
Ie iter
10ms
rosenbrock
griewank
himmelblau
Figure Optimizers offer distinct directions.
3

---

Published as a conference paper at ICLR 2026
Building this assumption, DOIT attempts propose dynamic optimizer switch method that leverages the strengths of different optimizers for distinct parameter states. Let o E O, À E A, and t E T denote the optimizer type (e.g., SGD), hyperparameter setting (e.g,, learning rate 0.1) and the training time (e.g., iterations), respectively. Then, the training process with dynamic optimizer switches can be defined by list of configurations C {cl, e², --.. ..,,"))) where et = (o', X f.). The
objective of DOIT is to find an optimal C that minimizes the following objective function:
C = arg co M.D,C)
(1)
where 0 is the initial model parameter state, [(-) is the loss of the trained model M under dataset D. Equation 1 represents dynamic optimizer tuning during training. When all ct share the same settings, aligns with the traditional training, which is described further in Section 4.1 For clarity, we fix all training times t E T to a constant T (i.c., 5 iterations) in the following sections.
3.3 ESTIMATING PERFORMANCE WITH SURROGATE MODELS
DOIT employs a Sequential Model-Based Optimization (SMBO) to address dynamic optimizer tun- ing, shown in Figure 5 Initially, DOIT trains model .M with random configurations to gather experience. It then constructs surrogate models G = 92..92..... 9m for each optimizer type O E O for ....... ..... J to guide configuration selection. By iteratively selecting configura- tions and updating surrogate models, DOIT achieves a dynamic optimizer switch training. In this section, we introduce DOIT's surrogate model through its selection, input, output, and initialization.
model with parameter B configuration cf (0,,
0g
training loss
2.1
OTO
-0-
-
-
Ha
6.1
6.4
6.1
1.6
0
11.0
4.0
11.0
Try Adam firs??
Aide dislon
Aupper
-1.
1.9
4.1
weighted random sampling Ron)
pvec. surrogate models
X
1.1
Ajower
5,7
5.7
6.1
surrogated selection (1 7lmil
iterations
4.5
5.1
6.5
Figure 5: The training process of DOIT.
Figure Examples of the optimization gain.
The selection of the surrogate model. DOIT adopts Gaussian process (GP) models (Schulz al.) 2018) as surrogate models for several reasons. First, compared to other models, GPs can efficiently train and continuously update the surrogate model as training progresses. Second, GPs provide uncertainty estimation for predictions (i.e., the variance), which is useful for guiding the optimizer selection (as detailed in Section 3.4). Third, as a powerful probabilistic model, GPs effectively construct the overall distribution based on known points, offering flexibility and interpretability.
The input of the surrogate model. At the beginning of each switch cycle, DOIT acquires the input for the surrogate model VEC!. The traditional surrogate model in SMBO uses the hyperparameter X' as its input. In DOIT, the input VEC* also includes a vector representing the parameter state O' to learn the impact under different parameter states. To reduce input size and cost, DOIT uses Principal Component Analysis (PCA) (Labrín & Urdinez, 2020) to compress the parameters layer by layer, lowering the training cost for the surrogate model. Additionally, DOIT selects only a subset of layers (e.g., the classifier layer with few hidden layers) as inputs for the surrogate model to minimize training costs. the case of PEFT, DOIT focuses solely on the trainable parameters (e.g.. the matrices A and B in LoRA (Hu et al. 2021).
The output of the surrogate model. contrast to the results obtained from training to convergence, DOIT emphasizes the "short-term" benefits each optimizer can achieve given the current parameter state. Therefore, the output of the surrogate model does not use the final loss or accuracy, but instead employs computed performance score E 1]. During switch cycle, DOIT performs iterations of training, resulting in a set of losses, denoted as I = {11 12.22,.........,, 4-1-1, loss before training. DOIT first calculates the loss variation Al; = max(U..4 .) for each iteration to get the average reduction μa and variance O To estimate the optimization performance of different configurations, DOIT combines the considerations of exploration (i.e.. variance a A and exploitation (i.e., mean (A) to calculate weighted score
L, and ly represents the
UA + ad A- However, such weighted score
4

---

Consideration of transferability. The training cost of DNNs is closely related to the initial model parameter state 0 In fine-tuning, closely related upstream and downstream tasks (i.e., high trans- ferability between the pre-trained model and the new task) are easier to train than those that are Published as a conference paper at ICLR 2026
overlooks the direction of variance. For instance, in Figure optimizers 01 and og have the same μa and σΛ. yet Og achieves a lower loss than or during training. A similar issue arises in the comparison between 02 with 01 and 03. To address this problem, incorporate boundary considerations into the performance calculation, including the upper bound AUPPer to lo-max(1) mmxx))))axx())x+ and lower bound
lo-min(4) LOWERR max()).min()))
Then, the final optimization performance score is defined as follows,
8 s= ==tnn, ( (US + DUPEER + ALWWER) + +aaa)
where a represents the weight for variance.
The initial weighted random selection. To obtain enough training experience for the construc- tion of surrogate models, DOIT trains with random configurations at the start of training. Though randomly selecting configurations for initial training can yield the necessary experience, DOIT em- ploys a weighted random initialization method to enhance the performance of the initial training. Specifically, DOIT maintains sampling weight wr] [0, for each type of optimizer oj and its surrogate model 9j. presenting the probability of being sampled. This sampling weight is initially assigned value of achieve random initialization. After completing the training with the cur- rent configuration, the corresponding optimizer's s sampling weight wi is updated to the normalized performance score shown in Equation
B). where wmin represents the minimal threshold.
w [j] = max( 2( Wiin))
3.4 SELECTING OPTIMIZERS WITH ACQUISITION FUNCTION
Although the calculated optimization performance can be used to select configurations directly, given the volatility of the loss and the complexity of model training, DOIT considers additional factors in the design of acquisition function, including variance, transferability, and the training process. In this section, we introduce these considerations in DOIT's acquisition function.
Consideration of variance. Benefiting from the advantages of the Gaussian process model, the surrogate model can provide both the mean score 8μ and an estimate of the variance 8,- Similar to traditional hyperparameter optimization methods, DOIT also incorporates trade-off between exploration and exploitation in the acquisition function as,
ACQ(Sns 8x) = 5y + Q87;
where a represents the weight for variance, consistent with the definition in Equation 2
dissimilar. Considering the idea that "a pre-trained model with lower transferability necessitates more substantial tuning adjustments", we use the model's transferability wi as the weight of the variance in the acquisition function, shown in the following equation,
ACQ(sp) So, wi) = S,, + (1 (1-w))as 88
(5)
The transferability We is calculated using two types of evaluation metries, including performance- based metric wp and distribution-bart metrics was First, the performance-based metric wp E [o, 1] is the test result (e.g., accuracy) which is directly tested with the pre-trained model without deep refining. Second, the distribution- based metrics analyze the distribution of the output vectors or labels to estimate the model's transferability, including LogME (You et al. (022)) and Leep (Nguyen et al. 2020). Equation (6) presents the definition of transferability weight.
Wi = Bwip + (1 B) Σ sigmoid(wg).
(6)
where w/ represents distribution- metrics and the weight for two kinds of metrics. A sigmoid function constrains distribution- metrics to [0, 1]. aligning them with performance- based metrics. The weighted sum then reflects the model's transferability to the current task: higher values indicate greater transferability, lower values the opposite.
5

---

Published as a conference paper at ICLR 2026
Consideration of training process. DOIT takes into account the differing needs in early and later training stages, specifically that "after the model becomes stable, smaller tuning adjustments are needed". As training progresses, the model gradually captures task knowledge, leading to stabi- lization of the training loss. At the later stages of the training. the target position on the parameter surface constrained within smaller range. In this context, optimizers with larger amplitudes may disrupt the tuning process. Therefore, the proportion of variance in the acquisition function should be reduced. Hence, we introduce periodic halving of the weight for variance information in DOIT as Equation ( where i represents the current iteration and n represents the halving period.
sigmoid 8p (1 -2-L4/n w.)so).
DISCUSSION
To further explore the characteristics of DOIT, we conduct discussion in this section. And further discussion about the memory overhead is presented in Appendix
4.1 ANALYZING THE DIFFERENCES BETWEEN DOIT AND HPO
Compare with AutoML. The optimizer can be viewed as hyperparameter in DNNs, and auto- matically selecting it is form of HPO and AutoML. However, vanilla training treats this as a static HPO problem, keeping the configuration fixed throughout. This static setting can be formulated as,
arg w0w.m DD.))
where c (o, X. denotes the configurations (as defined in Section 3.27 Compared to DOIT's dynamic tuning (i.e., Equation [ the static setting restricts the way model parameters are updated and the collaboration among different optimizers. In addition, though some hybrid methods (e.g.. SWATS (Keskar & Socher 2017 have been proposed, they are still limited to simple transitions rather than dynamic adjustments among various optimizers based on training states.
Compare with SMBO, DOIT adopts the idea of surrogate models and the acquisition function in SMBO (e.g., Bayesian Optimization), but there significant differences between them. While SMBO focuses solely on hyperparameter effects and performs static tuning, DOIT incorporates model parameter states and training progress estimate "short-term" gains, enabling dynamic tun- ing, Unlike SMBO's goal of finding fixed configuration, DOIT supports adaptive collaboration of optimizers throughout training to achieve better training performance.
4.2 ANALYZING THE ADVANTAGES DOIT
SGD
Adam
Accuracy. DOIT achieves DNN training with in- terleaving optimizers, enabling collaboration among multiple optimizers. This dynamic optimizer tuning not only integrates the strategies of different optimiz- but may also yield optimization path (i.e., the final trained model) that a single optimizer cannot achieve, resulting in higher accuracy. Figure ((a) pro- vides examples across three different functions, illus- trating that DOIT can discover optimization paths that a single optimizer cannot achieve. Similarly, the fi- nal model weights obtained from training for the same epochs on the cifarl0 dataset using different optimiz- ers show significant differences, as illustrated Fig- ure 7(b) Our proposed DOIT, which employs multi- ple optimizers, expands the search space of traditional training, raising the accuracy upper bound.
start
B
B
rasenbrock
griewank
himmelblau
(a) Optimization path
opt.
@ opt
3 opt
2 opt.
opt.
(b) Trained model weights (PCA)
Figure 7: Analyze of interleaving training.
Training efficiency and Memory Overhead. To evaluate the time overhead of DOIT, we use example. Table summarizes the FLOPs of DOIT's main components,
ResNet18 cifarl0 with detailed calculations provided in Appendix B Under typical settings (NEPOCH = 100, 7gz = 64, 20, m 103), the additional time introduced by DOIT is minimal (less than 1%). The

---

Published as a conference paper at ICLR 2026
optimizer overhead also remains within the normal range of the candidate optimizers. We further discuss convergence behavior and memory cost in Appendix B C
Table 1: Time cost analyze with ResNet18.
forward
gradient
optimizer
transferability
surrogate
acquisition
FLOPs
2.2x 1046
4.410016
0.2 2.0x 2.0x10 quite small
2.2x10 0.3%
5.3x 8.0%
1.0x1044
Proportion
33.3%
66.7%
1.5%
EXPERIMENTAL STUDY
To investigate the rationality of DOIT, first present overall results across various models and tasks, followed by case studies analyzing its switching behavior and selection preferences. In ad- dition, several independent experiments are presented assess key designs in DOIT. The main experimental settings are summarized below, with more detailed descriptions in Appendix D
Datasets. We evaluated DOIT on CV classification datasets (i.e., usps, mnist, st110, cifar10, ima- genet, imagenet-a) and 2 NLP datasets (i.e., mrpp and qqp), along with 3 additional tasks: machine translation (wmtl4), regression (eumite), and object detection (coco).
Models. We used 4 ImageNet pre-trained models available from PyTorch (i.c., ResNet18, ResNet152, MobileNet V2, and ViT) and pre-trained NLP model from HuggingFace (i.e., RoBerta). For other tasks, we used OPUS, TabNet, and Faster R-CNN, respectively.
Baselines. We used optimizers (i.e., SGD, SGDM, Adagrad, RMSProp, Adam, ASGD, AdamW, Nadam, and Adamax) and 4 hybrid optimizers (i.c., SWATS, Padam, AdaBound, and AGD), In addition, we also compared DOIT with optimizer using learning rate schedulers.
Setup. We set initial learning rates to 0..1,0.01, 0.001] and training epochs to 100. For DOIT, we used Pini 50, switch cycle of = 25 iterations, and an optimizer space of [SGD, SGDM, Adagrad, RMSprop, Adam]. Each experiment was repeated 5 times with different seeds on a Linux server with a 128-core 2.6GHz Intel Xeon Platinum 8358 CPU and 512GB RAM.
5.1 OVERALL PERFORMANCE OF DOIT
We first present the comparison results between DOIT and commonly used methods classification tasks, including full training and PEFT. We then broadened the evaluation to additional models and tasks, further comparing DOIT with other approaches demonstrate its effectiveness.
Table 2: Test accuracy (%) of full and PEFT training with ViT and RoBerta.
full training st100
PEFT
usps
mnist
cifar10
imagenet
usps
stl10
mrpe
SGD
97.46
97.83
97.53.1mm
94.42 78
97.75
85.21 joss
82.13
SGDM
97.68+0.1
99 65
96.6 =6.6
97.58=0m
78. 36,36=a12
95.67+0.14
98. =0.10
85.54=0m
83,30=06
Adagrad
93,40+eu
98. 24+ase
78.66-254
60.95+om:
77.59=00
95.37+ea:
98. =4
83.47+ 84.94+039
RMSprop
95.25
98. 14
88 28.622445
78.09
24.01
94.644055
97. 97.91 2207
84.09 2m
82.09
Adam
13.2
Lova
82.26
75.23
73.96_mm
Lown
98.36
86.52.em
82.27 1871
S SWATS
94,00+123
98. 73+03
88.03=044
66.
76.93=a1>
95.12+021
98 98.38=0100
86.27=ox2
50.79=aal
Padam
97.58aam
99.661a0t
90.8 gaow
96.03.amm
77
95.72.0.7
18.381010
80.64
23.0
AdaBound
87,64
97.54
86.33=239
70.91 +4.35
75.93=013
95.42+a11
98. 201
68.38
78. 64
ours
97.81491
99.71
98.21
98.04=990
79.98=08)
99.01 96.12+4.
87.99
85.57
Experiments on full training and PEFT. We first compared DOIT with optimizers from its search space and hybrid methods on classification tasks, as shown in Table 2 For CV tasks, we used ViT (head fine-tuning (Poojary & Pail 2019) for PEFT), and RoBerta with LoRA (Hu et al. 2021) for NLP. Experimental results show that DOIT often achieves higher accuracy (i.e., 1%-3% improve- ment) and lower variance. Meanwhile, existing hybrid methods perform poorly on some complex datasets (c.g., qqp), whereas DOIT adapts effectively to different datasets. To investigate the con- vergence of DOIT, recorded the convergence time (i.e., the change of loss is less than 1 x 10-1
7

---

Published as a conference paper at ICLR 2026
in 10 consecutive epochs) and loss trajectories of each method, shown in Figure and Figure The results show that DOIT achieves faster convergence in terms time and iterations. Additional results provided in Appendix E
static optimizer
bybrid optimizer
OUTS
SGD
SGDM Adagrad
RMSprop
Adam ours
1000
sa.10
10
101o
convergennc
3400
101
2500
1600
10
training loss
validation (%)
SGD
SGDM
Adagrud
RMSprop
Adam SWATS
Padam
AdaBound
ouri
25
50
50
100
training epoch
trinnng epoch
Figure 8: Convergence time on PEFT.
Figure 9: Loss and accuracy line on cifar/0.
More experiments. addition to the experiments above, we conducted further evaluations on wider range of tasks and models, and compared DOIT against more baselines. (i) More complex dataset: we conducted experiments complex imbalanced dataset imagenet-a (Hendrycks et al.)
2021] as shown in Table 3 DOIT's dynamic adaptation to critical saddle points enhances perfor- mance on complex problems, resulting over 2% improvement in top-1 accuracy. (ii) More
models, tasks, and baselines: Table 4 presents results on additional models, tasks, and baselines, where DOIT con- sistently achieves the best performance. More experimental results and detailed settings can be found Appendix E (iii) Detailed time cost analyze: Figure 10li- il-
training
optimizer
selection
transferability
SGD
SGDM
221
3.6
1.523
26
Adagrad
215
1,621
RMSprop
219
6..1
1,50
lustrates the time consumption of differ- ent components across various methods. Compared to static optimizers, the addi- tional time cost introduced by DOIT is minimal (less than 0.5%), which aligns with our analysis Section 4.2
Adam
216
1.564
68
0.04
0.04
0.8
205
215
225
1,400
1,500
1,600
1,700
Lime cost (sec.)
time cost (sec.)
Figure 10: Time cost of each step.
Table 3: Test accuracy (%) of the full training on imagenet-a.
SGD
SGDM
Adagrad
RMSprop
Adam
SWATS
Padam
AdaBound
ours
acc
15.24
16.19
5.20
4.13
3.53
3.68
16.31
4.33
18.47
acc@83
30.74
31.20
13.12
10.42
9.87
8.94
31.76
11.23
33.83
ucc@5
38.29
39.68
17.49
17.52
13.83
12.55
40.01
15.46
41.32
Table 4: Experiments with more models, tasks, and baselines.
(a) more models
(b) more tasks
(c) more baselines
stl00
wmtl4 BLEU
cunite R2
usps
mnist
accuracy
metric model
AP50
ASGD
95.83
99.25
ResNet18
ResNet152 MobileNet
OPUS
TabNet
RCNN
AdamW
95.62
98.97
SGDM
91.87
96.66
92.14
SGD
27.91
71.81
57.44
Nadam
95.32
99.18
Adam
91.02
96.
91.31
SGDM
28.12
70.32
$7.26
Adamax
96.06
99,32
SWATS
92.02
96,66
92.77
Adagrad
27.14
71.22
56.46
AGD
95.96
99.19
Padam
92.36
96.51
91.69
RMSprop
27.41
67.88
$3.08
stepL.R
91.31
94.72
AdaBound
92.23
95.44
91.39
Adam
28.24
69.83
56.46
cosL.
95.80
99.46
ours
93.56
OuTs
28.68
73.12
59.21
96.81
99,51
8

---

Published as a conference paper at ICLR 2026
5.2 ANALYZE THE SWITCHING PROCESS OF DOIT WITH CASE STUDIES
To further investigate the patterns and preferences of DOIT in optimizer selection, we conducted several case studies. (i) When does the optimizer switching occur? The left plot of Figure [1 shows the training process of DOIT on a simple dataset, hymenopiera, where it switches only be- tween SGD and SGDM. After random initialization, DOIT performs switching operations when decrease in the convergence speed of the optimizer (Point A) or detects a local stable state (Point C). In addition, during tuning. DOIT may also undergo temporary switches adjust the optimiza- tion state (Point B). (ii) What is DOIT's selection preference like? The right plot of Figure [1] presents DOIT's switching probabilities and selection patterns across different training stages (with switching limited to SGD and Adam). In summary, we have identified the following characteristics:
"As training progresses, the selection of optimizers becomes more stable (fewer switches)." Figure b) illustrates the probability of optimizer changes at different training stages. It shows that the likelihood of switching optimizers gradually decreases over time.
"In the early training phase, there is tendency to select optimizers that converge faster (e-g., Adam), while in the later phase, DOIT tends to select more stable optimizers (e.g,, SGD)." As shown in subplots (c) and (d), DOIT demonstrates a stronger preference for SGD the later training stages, which aligns with the findings of SWATS (Keskar & Socher 2017).
"DOIT's selection preference is more apparent in a large dataset." Comparing subplots (c) and (d), DOIT shows clearer selection preferences on mnist than usps. Larger datasets provide more experience for surrogate model training. allowing for more stable selection of specific optimizers.
DSG SGDM
switch line
switch ratio
switch detail
training loss
2.4
80
(b)
Adam
2.0
60
(c)
SGD
1.6
init
maist
SGDM
20
(d)
ynnist
SGD
25%
100%
0%
100%
training epoch
training slage
training slage
Figure 11: The training loss line of the case study.
5.3 INDEPENDENT EXPERIMENTS
Moreover, we conducted independent experiments to further analyze the effectiveness of DOIT. We out- line the main conclusions, with further details avail- able in Appendix F (i) Varying search space. Fig- ure 12 shows the impact of expanding the search space (i.e,, more optimizer types and hyperparam- eter options) on DOIT's performance. strates that DOIT continues perform well in the enlarged search space, with only a minimal increase in time cost. (ii) Ablation experiment of key de- signs. To assess the effectiveness of DOIT's key design components. we conducted series of abla- tion studies. These include comparisons on the opti- mizer selection strategy (DOIT vs. random switch- ing and periodic replacement), acquisition function design (DOIT vs. versions without transferability weight W: training process halving), initial selec- tion strategy (DOIT's weighted random vs. uniform random), and parameter compression method (PCA used in DOIT vs. random projection and UMAP). Experimental results demonstrate that each com- ponent of DOIT's design contributes to improving both convergence speed and model quality. (iii)
baseline
ours
ours(time)
96.5
300
8ost accuracy
200
tine (xec.)
95.5
100
demon-
95.0
SGID
Adagrad
RMSprop
ASGO
Adam
AdamW
NAdam Adamax
sost aecurcc (%)
97.2
200
96.7
100
96.2
3
additional HP types
Figure 12: Results with varying space.
9

---

Published as a conference paper at ICLR 2026
Hyperparameter analysis. We also performed an experimental analysis on the setting of hyperpa- rameters in DOIT, including the initial step ninns, switch step size т, number PCA components, and selected compression layers. The results show that a small initial step (e.g., only 10 for small dataset usps), switch step size (10% of an epoch) and PCA components (e.g.. 2) can help achieving good accuracy with limited time cost.
6 CONCLUSION
Optimizers play pivotal role in training deep neural networks. However, current training ap- proaches (i.e., using a static optimizer or simple hybrid methods) struggle to adapt to the evolv- ing optimization landscape across different training states. This lack of adaptability can degrade both convergence and model quality. To address this limitation, we propose novel method called Dynamic Optimizer Interleaving Training (DOIT) method, which dynamically selects suitable opti- mizers based on the training states. During training, DOIT leverages surrogate models to estimate the performance of various optimizers given the current parameter state. It then combines these estimates with transferability assessments and processes information through a special acquisition function predict potential optimization gains. At each training state, DOIT selects the optimizer with the highest score for training. Experiments results show that DOIT achieves higher accuracy (i.e., 1% 3% improvement) and faster convergence (i.e., 2% 10% faster).
Limitations and future work This work focuses on dynamic optimizer switching during training, particularly regarding the types of optimizers used. Our method does consider the inheritance of internal states (e.g., momentum) when switching optimizers. While prior studies (Chen et al., 2018 Yue et al. 2023) explored inheritance between two optimizers, extending it to many candi- dates greatly iccrasses complexity. As discussed in Appendix G simple inheritance does not yield consistent gains and may introduce instability, so robust cross-optimizer state transfer remains an important direction for future work.
ACKNOWLEDGMENTS
This work is supported by the National Natural Science Foundation of China (Grant No. 62376099) and the Natural Science Foundation of Guangdong Province (Grant No. 2024A1515010989)
REPRODUCIBILITY STATEMENT
All models, datasets, and baseline methods used in this work are publicly available. Detailed de- scriptions of experimental settings, hyperparameter configurations, and training procedures are pro- vided in the Experiments section and further elaborated in the Appendix. To ensure full repro- ducibility, we will also release the complete implementation and scripts for all experiments.
REFERENCES
Steven Adriaensen, André Biedenkapp, Gresa Shala, Noor Awad, Theresa Eimer, Marius Lindauer, and Frank Hutter. Automated dynamic algorithm configuration, 2022. URL htttts:://raak erg/abs/ 2205. 13881
Sercan O. Arik and Tomas Plister. Tabnet: Attentive interpretable tabular learning, 2020. URL ptttss://arxvv. htrg//bbs/900 07442
Francesco Barbieri, Jose Camacho-Collados, Leonardo Neves, and Luis Espinosa-Anke Tweet-
eval: Unified benchmark and comparative evaluation for tweet classification. arXiv preprint 12X1::2010.12421. 2020.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research (JMLR), 13(2), 2012.
André Biedenkapp, Nguyen Dang, Martin S Krejea, Frank Hutter, and Carola Doerr. Theory- inspired parameter control benchmarks for dynamic algorithm configuration. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), pp. 766-775, 2022.
10

---

Published as a conference paper at ICLR 2026
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow. Philipp Koehn, Johannes Leveling. Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale Tamchyna. Findings of the 2014 workshop statistical machine trans- lation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 12- 58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. URL http: httk:::/www..ccc www.aclwwb...... weapggganthoooggo //tp////1/// ////14/W14--1 3
Chih-Chung Chang and Chib-Jen Lin. Libsvm: a library for support vector machines. ACM Trans- actions Intelligent Systems and Technology (TIST), 2(3): 2011.
Bo-Juen Chen, Ming-Was Wei Chang, et al. Load forecasting using support vector machines: A study on eunite competition 2001. IEEE transactions on Power Systems, 19(4):122 2004.
Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXiv:1806.06663, 2018
Adam Coates, Andrew Ng, and Honglak Lee, An analysis single-layer networks in unsupervised feature learning. In Proceedings of the fourleenth international conference artificial intel- ligence and statistics (AISTATS), pp. 215-223, JMLR Workshop and Conference Proceedings, 2011.
Gaole Dai, Wei Wu, Ziyu Wang, Jie Fu, Shanghang Zhang, and Tiejun Huang. HUB: Enhancing learned optimizers via hybrid update-based strategy, 2024. URL htttps://penreviiew feetfooru?iddie HdELJ1s
Christian Daniel, Jonathan Taylor, and Sebastian Nowozin. Learning step size controllers for ro- bust neural network training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Dorottya Demszky, Kelvin Guu, and Percy Liang. Transforming question answering datasets into natural language inference datasets, 2018. URL [tLLittts////xxii/ hwtp://wwww.ggabb////bbbb// 02922
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the International Workshop on Paraphrasing, 2005.
Timothy Dozat. Incorporating Nesterov Momentum into Adam. In Proceedings of the 4th Interna- tional Conference on Learning Representations (ICLR), pp. 1-4.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research (JMLR), 12(7), 2011.
Matthias Feurer and Frank Hutter. Hyperparameter optimization. Automated machine learning: Methods, systems, challenges, pp. 3-33, 2019.
Alex Graves. Generating sequences with recurrent neural networks. 13Xiv:1308. 0850,, 2013.
arXiv preprint
James Harrison, Luke Metz, and Jascha Sohl-Dickstein. A closer look at learned optimization: Stability, robustness, and inductive biases, 2022. URL
11208
Esraa Hassan, Mahmoud Shams, Noha A Hikal, and Samir Elmougy. The effect of choosing optimizer algorithms to improve computer vision tasks: a comparative study. Multimedia Tools and Applications (MTA), 12111)1655,,11633,, 2023.
Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-art. Knowledge- based systems (KBS), 102:22 2021.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples, 2021. URL http:/tttt:///iiiieee
abs/ /
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 2XXiv:210600666: 2021.
11

---

Published as a conference paper at ICLR 2026
Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions pattem analysis and machine intelligence (TPAMI), 16(5):550 554, 1994.
Daniel Jiwoong Im, Michael Tao, and Kristin Branson. An empirical analysis of the optimization of deep network loss surfaces. arXiv preprint 1612:: 2016.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd. arXiv preprint 1rXiv:1712.07000 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /4/2.4980 2014.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https://arxiv org/ wwtp/////////ooo/oeoo.oooe..eese 1412. 6980
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Caterina Labrín and Francisco Urdinez. Principal component analysis. In R for Political Data Science, pp. 375-393. Chapman and Hall/CRC, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEF, 96(11):2278-2322) 1998.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer vision-ECCC 2014: 13th European conference, zurich, Switzerland, September 6-12 2014, pro- ceedings, part 13, pp. 740-755. Springer, 2014.
Dong C Liu and Jorge Nocedal, On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1-3):503-55 1989.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
//arxiv. wttp://wwwrggbb///1ll.ooo
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic 2019.
bound of learning rate. arXiv preprint
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factoree approximate curvature. In International conference on machine learning (ICML.. pp. 2408-2417, 2015.
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning (ICML), pp. 7294-7305. PMLR, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Yang, Zach Veti,o. Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performanee deep learning library. In Neural Information Processing Systems (NeurIPS), 2019. URL https: htapps://ppl www.ntisschotooooooe.e.cc
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-leam: Machine learning in python. The Journal of Machine Learning Research (JMLR), 12:2825-2830, 2011.
B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Control Optim., 30(4):838 855, July 1992. ISSN 0363-0129. doi: 10.11370033046. URL.
https: / oro.orgggooo 100 1111//..111100000400
Ramaprasad Poojary and Akul Pai. Comparative study of model optimization techniques in fine- tuned enn models. In 2019 International Conference on Electrical and Computing Technologies and Applications (ICECTA), pp. 1-4, IEFE, 2019.
12

---

Published as a conference paper at ICLR 2026
Isabeau Seemont-Schwarz Jaroslav V'itkru, and Jan Feyereisl. A simple guard for learned op- timizers. ArXiv, abs/2201.12426, 2022. URL https://appi semanticsccooiii .org/
Corpus_D:24643033
Guannan Qu and Na Li. Accelerated distributed nesterov gradient descent. IEEE Transactions on Automatic Control, 65(6):2566-2535) 2019.
Esther Kim Quora. (2017). Question Pairs Dataset, Version 2. Retrieved May 1, 2024 from https:
7/www. kagg com cttp://aaaattets//ogrraaaaaaaaaa
dataset.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks, 2016. URL https::/aaxxiv.or///aa
01497
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statisties, pp. 400-407, 1951.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint NEti:0/609007444 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 15(3)2211-252 2015. doi: 10.70/ 1263-015-0816-y.
Eric Schulz, Maarten Speekenbrink, and Andreas Krause. A tutorial on gaussian process regression: Modelling, exploring, and exploiting functions. Journal of Mathematical Psychology, 85:1-16, 2018.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1); 148-175, 2015.
Derya Soydaner. A comparison of optimization algorithms for deep learning. International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI). 34(13):2052013, 2020.
Ruo-Yu Sun. Optimization for deep learning: an overview. Journal of the Operations Research
Suciety of Chinu, 8(2):249-294, 2020.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initializa- tion and momentum in deep learning. In International conference muchine learning (ICML), pp. 1139-1147. PMLR, 2013.
Jörg Tiedemann and Santhosh Thottingal. OPUS-MT - building open translation services for the world. André Martins, Helena Moniz, Sara Fumega, Bruno Martins, Fernando Batista, Luisa Coheur, Carla Parra, Isabel Trancoso, Marco Turchi, Arianna Bisazza, Joss Moorkens, Ana Guerberof, Mary Nurminen, Lena Marg, and Mikel L.. Forcada (eds.), Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pp. 479- 480, Lisboa, Portugal, November 2020. European Association for Machine Translation. URL https://aclaathh/
org/ 2020 eamt-l. 61/
Anna Vettoruzzo, Mohamed-Raan Ratik Bouguelia, Joaquin Vanschoren, Thorsteinn Rögnvaldsson, and KC Santosh. Advances and challenges in meta-learning: A technical review. IEEE Trans. Pattern Anal. Mach. Intell., 46(7):77634477, July 2024. ISSN 0162-8828. doi: 10.1109/TPM..20224 10. 1109/10PPPP 11199/PAMMI 2024. 3357847
3357847. URL httltttts://ooooo
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/1/W11--555 URL https://ach antho wwcchhoooy.orgoo www.oogg///1--502///0
13

---

Published as a conference paper at ICLR 2026
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. Marilyn Walker, Heng Ji, and Amanda Stent (eds.). Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp- 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguisties. doi: 20.18653/v1/N18-11
URL
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State- of-the-art natural language processing, 2020.
Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), (511)))0116066661107777 2021.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre- trained models for transfer learning. In International Conference Machine Learning (ICML), pp. 12133-12143, PMLR, 2021.
Yun Yue, Zhiling Ye, Jiadi Jiang, Yongchao Liu, and Ke Zhang. Agd: an auto-switchable optimizer using stepwise gradient difference for preconditioning matrix. Advances in Neural Information Processing Systems (NeurIPS), 36:45812 2023.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvorekk, Xenophon Pa- pademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in ob- served gradients. Advances in Neural Information Processing Systems (NeurIPS), 33:18795- 18806, 2020.
A OVERALL AL.GORITMM AND PSEUDOCODE DOIT
For ease reading, we provide detailed explanation key notations used in this paper for DOIT in Table 5
Table 5: The description of notations used DOIT.
notation
description
switch step r
the number of iterations for single switch cycle. the number of switch cycles for the initial phase. the sample weights for the initial phase transferability weight which is calculated by the performance- based metric we and distribution-
init step Nimi
selection weights We
transferability weight wa
metric wid
In this section, we provide overview of DOIT with the pseudocode in Algorithm I For ease of understanding, we only switch the types of optimizers in the description while keeping the hy- iterations). Before the model's training,
perparameters and training time constant (i.e., t DOIT first calculates the transfcrability weight to assist in the subsequent selection of optimizers (Line Then, a randomly selected initial optimizer oj used for model training and loss cal- culation (Lines 6) When the number of iterations meets the time constant DOIT calculates the performance score s based on all losses within switch cycle losses (Line g) Subsequently, the current optimizer oj is updated by the surrogate model gj using the optimization gain and the model features a calculated at the end of the previous round (Lines 10). Based on the results from weighted random selection surrogate model selection, DOIT obtains the suitable optimizer for the next training stage (Lines I0 15) and continues this process iteratively until the final training results 2h are achieved.
14

---

Published as a conference paper at ICLR 2026
For the selection of the suitable optimizer, DOIT employs two types of methods: the weighted random selection and the surrogate model selection for the following steps. In the initial steps (i.e,, n4 < r), DOIT uses the optimization gain to update the sampling weight w, for randomly select configurations (Lines To 13). After obtaining enough training results (i.e., ni 2 т), DOIT utilizes the trained surrogate models select the configurations used for the following training (Lines [3 15 The configuration with the highest score (i.c., Equation is selected for the next training iteration.
Algorithm 1: Basic framework of our proposed DOIT.
Input:
Training Dataset D (Drruins Drest J. optimizers O 1.,,..., Om ). the model for training M with initial parameter state 6%, the init steps Dau., the switch step size 7, the training epochs
Repoch.
initialize: n2 +- 0, n + 0, 921-92..--mmm w, + [1]1,..,m] < RandomSelect(m),
losses + ||. + CompressModel(MM,
/+ calculate the transferabile lity weight before the training W f Calculate TranlltteTranffffrbblttttti Weibhlt M, Dirain) (as Equation
/+ performance evaluation and optimizer switch during FT
for in [1, Repochh do
for BATCH in Dirain do
n - n + 1, + TrainModel 0- BATCH, o;) [ + CalculateLoss(MM 0", Bor.aa/.MM...BBAAAA....sess..ppa
if nr then
Calebaaccoptmmizziiootaaaansssss
(as Equation
01 +- UpddateSurogalcMMllee((1] v,
F < CompressModel(MM, 0")
// init steps with weighted randon selection
if < Rin then
wr[j] < SagbateSaappeeeiihht'
(as Equation
13
j + WeightodRandonSsttt(((..))
following steps with surrogated selection
else
15
j + SurrogatedSeloct((, wr, 9,
16
- n4 lossess +
17
end
18
end
Output: the trained model M with parameter state d
B DETAILS ABOUT THE ANALYSIS OF TRAINING EFFICIENCY
Here, we provide detailed description of the time cost calculation mentioned in Section 4.2 based on the following analysis scenario:
Model:
ResNet18, who contains approximately W = 1.17 x 10" parameters. Its forward pass requires around trw 3.66 10 FLOP and we estimate the cost of backpropagation and gradient updates as twice that of the forward pass, yielding tpp ~= 7.32 x 10 FLOPs.
cifar10, whose feature dimensions D as 3 x 103, instance number N class number K = 10.
Dataset:
104 and
Hyperparameter setting: We the epoch number ПerCCи 100, batch size fuz = 64, switch- ing iteration number 7 20, and the optimizer search space of DOIT m 1033
For vanilla training, the total time cost primarily consists of three components: forward propagation fiwwards backward propagation fackwand. and parameter updates fupdate- The FLOPs of forward and backward propagation are related to the number of instances and training epochs, and are given as
follows,
Lorward /rw N THPACH 2 2.2 10
thackward = tpp N NEPACH ~ 4.4 1026.
htt http:///ccaelallsaaaaaaccsoollls
vd_.e..
html
15

---

Published as a conference paper at ICLR 2026
In addition, the FLOPs for parameter updates are dependent the optimizer type, the number of model parameters W, and the batch size fzz. Different optimizers require different computational steps Nupe for parameter updates. For example, in SGD, each parameter update requires 2 com- putations, whereas Adam requires approximately 18 computations. Using these two optimizers as examples, the FLOPs for parameter updates
follows:
N
Tupdate = Nopl X W
REPOCH R 2.2 10 ~ 2.0 10
TozZ
For DOIT, the FLOPs for forward foowwrdd and backward propagation Backward are the same as those in vanilla training, while the parameter update FLOPs Kupdate is in the range of the value in the optimizer space. Thus, we have,
(orward fooward Rs 2.2 x 10
Bbackward hackward 4.4 10
Cupdatee (upasttu mwimmnntttttt (Fupdas)) = [0.2 10t3,
10
Moreover, compared to vanilla training, DOIT mainly introduces additional time overhead from the following three components,
transferbility assessment: The assessment includes the computation for two distribution-ban metrics (i.e., LEEP and LogME) and one performance-based metric. Among them, the time cost for the LEEP and performance-bact
metrics is equivalent to single forward pass (Nguyen et al. 2020), while the computational complexity of LogME is flogme KD + NKD + D + Ro 3 x 10 (You et al. 2021). Since the transferability evaluation is performed only once before training, its FLOPs is equal to,
iransfer fFw X N flogme ~ 2.2 1023
surrogate model updating: This time consumption includes the PCA compression of the selected parameters and the updating of the Gaussian process model (primarily driven by the computation of the covariance matrix). The time complexity of compression and updating is O se D') and O(na)) where Wae 10 represents the number of selected parameters (i.e., only the last layer), D' R 100 represents the number of PCA components, New = NAgPCCH 7X the total switching operations in the tuning process, and na the number of observation points in surrogate model. In our computation, we set to the use of multiple surrogate models for different optimizers and varying update times. Then we can calculate the FLOPS as follows,
Rs 4.7x 10 represents
N.w. although the actual value is smaller due
Kaurogaate = New x (W + n = 5.3 x 10
acquisition function calculation: The computation of the acquisition function is mainly deter- mined by the performance evaluation of the surrogate model. Specifically, the prediction com- plexity of Gaussian processes for single candidate configuration is On Then the FLOPs of acquisition function calculation is,
acquisitiin = New X m X n ab 1.0 x 14
Based the above analysis, the additional FLOPs introduced by DOIT are quite small compared vanilla training (less than 1 1%). While factors such as hardware, GPU parallelism, and optimizer type can influence the exact runtime, the overall increase training time is consistent with the FLOPs conclusion. Moreover, due to careful optimizer selection, DOIT often achieves faster convergence.
As for the convergence behavior, although DOIT performs optimizer switching during training, its overall convergence behavior is guaranteed by the underlying optimizers. DOIT does not modify any optimizer's update rule and instead selects from standard optimizers at different stages, so the convergence behavior aligns with that of the chosen optimizer at each stage. Moreover, the acqui- sition function encourages DOIT to favor more stable optimizers in the later phases of training (as shown in Sec. 5.2), ensuring stable and reliable convergence.
C MEMORY OVERHEAD ANALYSIS
We analyze the additional memory overhead introduced by DOIT to address concerns regarding optimizer states and surrogate models. Overall, the overhead is small, approximately 13% of
16

---

Published as a conference paper at ICLR 2026
R parameter size, and remains consistent across models and datasets. The additional mem- ory overhead in DOIT mainly comes from: (1) storing the history of compressed parameter vectors and performance scores (Mrrec), and (2) maintaining Guassian process surrogate models (Mrr.). Following the analysis settings in Appendix
B with switch steps New 2 2.7 x 10 and PCA dimension D' 100, the history cost is M New D 2.7 x 10 For K candi- date optimizers, each surrogate model stores approximately K Now points, so kernel memory costs M = K ( a..)) = 1.46 x 10 Together, these account for roughly 12.5% of ResNet18's parameter size (less than 3% of total training), and are constant across model sizes and datasets, making the memory overhead acceptable in practice. In addition, regarding optimizer states, DOIT does not maintain states for multiple optimizers simultancously. Instead, optimizer states are re- set at each switch, so only one optimizer's state is active at any time, resulting in memory usage comparable to a single optimizer.
D DATASETS, MODELS, BASELINES, AND SETTINGS IN EXPERIMENTS
In this section, we provide detailed description of the datasets, models, baselines, and experimental settings used in our experiments.
Datasets. We used CV classification datasets (i.e., usps, mnist, stl10, cifar10, imagener, imagenet- a) and 5 NLP datasets (i.e., mrpc, ric, qqp, gnli, and mnli), along with 3 additional tasks: machine translation (wmt/4), regression (eunite), and object detection (coco). All the datasets were loaded from PyTorrh (Paszke et al.) 2019), LibSVM (Chang & Lin 2011) or HuggingFace (Wolf et al., 2020
usps (Hull 1994) (from PyTorch): classical digit dataset automatically scanned from envelopes by the U.S. Postal Service containing a total 9,298 16x16 pixel grayscale samples, which in- cludes 10 classes of figures.
mnist (LeCun et al. 1998) (from PyTorch): a handwritten digits dataset with 28x28 grayscale figures, which has a training set of 60,000 examples and a test of 10,000 examples.
stl10 (Coates et al. 2011) (from PyTorch): 10-classes 96x96 color figure dataset, which has 500 training images and 800 test images per class. The dataset is inspired by the cifar-10 (Krizhevsky et al. 2009) but with some modifications.
cifarl0 (Krizhevsky et al. 2009) (from PyTorch): 10-classes 32x32 color figure dataset, which has 5,000 training images and 1,000 test images per class.
imagenet (Russakovsky et al. 2015) (from PyTorch):
The ImageNet 2012 dataset (also known as ILSVRC 2012) classification subset contains 1,000 categories. The training set is approximately 137 GB, while the validation set is around 6 GB. Each class in the training contains about 1 1,300 images, totaling approximately 1.3 million images. Each class in the validation set contains 50 images, for total of 50,000 images.
imagenet-a (Hendrycks et al., (02T) (from HuggingFace): includes real-world, unaltered, and naturally occurring examples that were misclassified by the ResNet model. The dataset consists of 7,500 adversarially filtered images, which can severely impact the performance of machine learning models.
mrpc (Dolan & Brockett] 2005) (from HuggingFace): Microsoft Research Paraphrase Corpus, which consists of 5.8k sentence pairs that were automatically extracted from online news sources. The sentence pairs have been annotated by human raters to indicate whether the sentences within each pair are semantically equivalent.
rte (Wang et al. 2018) (from HuggingFace). the recognizing textual entailment datasets come from a series of annual textual entailment challenges. Examples constructed based news and Wikipedia text. The authors of the benchmark convert all datasets to a two-class split, where for three-class datasets, they collapse neutral and contradiction into not entailment, for consistency.
qdp (Quora) (from HuggingFace): the Quora Question Pairs dataset, which consists of over 400,000 pairs of questions. Each question pair is annotated with binary value indicating whether the two questions are paraphrases of each other.
qnli (Demszky et al. 2018) (from HuggingFace), Stanford Question Answering Dataset, which is a questionn anwwering dataset consisting of question- -paraginp pairs, where one of the sentences in
17

---

Published as a conference paper at ICLR 2026
the paragraph (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator).
mnli (Williams et al. 2018) (Multi-Genre Natural Language Inference Corpus, a crowdsourced collection of sentence pairs with textual entailment annotations.
wmi14 (en-de) (Bojar et al. 2014) (from Hugging[ace)): is a collection of datasets used in shared tasks of the Ninth Workshop on Statistical Machine Translation. We used the German-Englist translation dataset, which contains 4.51 million training samples, 3,000 validation samples, and 3,000 test samples.
eunite (Chen et al.
2004) (from LibSVM): the regression dataset for electricity load forecasting.
2014) (from HuggingFace), the object detection datasets COC02017, which contain 135K images for training and 5K images for validation.
coco (Lin et al.
Models. For the pre-trained models, we used 4 ImageNet pre-trained models available from Py- Torch (Paszke et al. 2019) (i.e., ResNet18, Res MobileNet V2. and ViT) and 4 pre-trained NLP models (i.e., RoBerta, GPT-2, QWen2-0.5B. and LLaMA-3 -...------1B from HuggingFace (Wolf et al. 2020) which trained on 124M tweets from January 2018 to December 2021, and finetuned for sentiment analysis with the TweetEval benchmark (Barbieri al., 2020). It can be found that among the downstream tasks, st/10, mrpc and qqp are relatively close to the upstream task, and usps and mnist have certain correlation with the upstream task. We selected downstream datasets with varying degrees of relevance to the upstream task, to better analyze the performance of the proposed method in different scenarios. For other tasks, we used OPUS (Tiedemann & Thottingal 2020) from HuggingFace, TabNet (Arik & Pfister (2020) from pytorch-tabnet libray, and Faster R-CNN (Ren et al. 2016) from PyTorch, respectively.
Baselines. In the overall experiment, we compared 9 single optimizer methods (i.e., SGD (Robbins) & Monro, 195T). SGDM (Sutskever et al.) 2013), Adagrad (Duchi et al. (20TT), RMSProp (Graves) 2013). Adam (Kingma & Ba, 2014), ASGD (Polyak & Juditsky, 1992), AdamW (Loshchilov Hutter, 2019), Nadam (Dozat), and Adamax (Kingma & Ba 2017)) hybrid methods (i.e.. SWATS Keskkrr& & Socher 2017), Padam (Chen et al., 2018), AdaBound (Luo et al., 2019). and AGD (Yue et al. (003)), and the proposed DOIT. The single methods are all from the Py PyTorch im- plementation, and except for the learning rate being set in 1,,0.0,,,.00000 and the epoch number being set to 100, the other hyperparameters are set to the default values in PyTorch. Additionally, for the hybrid methods, we installed and used the original implementations from the authors via Github and PyPI. The hyperparameter settings were kept at their defaults, except for the epoch number, which was adjusted be consistent with the other methods. For DOIT, we used fimi = 50, switch cycle of 25 iterations, variance weight = 0.1, weight for transferability assessment / = 0.8, halving period
Rgrocu/44 44 and an optimizer space of [SGD, SGDM, Adagrad, RMSprop, Adam]. In addition, due to the large size of the imagenet, we set the number of epochs to 10 and the batch size to 256. As for the Gaussian surrogate model, we used the Caussian from scikit-learn (Pedregosa et al., 2011)) with its default settings.
E MORE DETAILS AND RESULTS ABOUT OVERALL EXPERIMENTS
In this section, we provide additional experimental details and results in Section 5.1
Overall performance. In addition to the test accuracy results of the ViT and RoBerta models on the we also performed full training on ResNet18 and recorded the Fl score of RoBerta PEFT. The results are shown in Table
classification dataset shown in Table 2
Convergence analysis. In addition to the loss lines for full training cifar10 in Figure 9 we also recorded the results on usps and st/10, as shown in Figure [3] For ease of observation, the baselines with significant fluctuations not displayed in the figure. It can be observed that DOIT shows faster convergence speed compared to the vanilla method.
More models, tasks, and baselines. Table 4(a) presents experimental results with more CV models under head fine-tuning. We provide extended results in Table 7 In the experiments, all methods have run 100 epochs without early stopping. In addition, Table 4(c) reports only the best results obtained with training using a scheduler, specifically those achieved with SGDM. Moreover, we further evaluate DOIT on additional NI.P models and tasks. Specifically, the experiment includes
18

---

Published as a conference paper at ICLR 2026
Table 6: Test accuracy (%) of full training ResNet18 and Fl score (%) of PEFT training RoBerta.
full training
PEFT
usps
mnist
stl10
cifar10
usps
mnist
mipe
qqp
SGD
99.29=6.0
86,9420.9
80.73+04s
94,42=0a1
97.75
87.21 +0.32
75.19
SGDM
05.83.i.0
99,47
86,9910.6
81.64100
95.67
98.37
86.2710.1
75.27
Adagrad
96.00=a.4
99,40-00
83.38=789
80.57:0.14
95.37=0a1
98.34
87.29#046
73.28
RMSprop
95.30
99.13 13
69,91
71.92+041
94.64_0.sa
97.91 2000
89.242n74
74.09
Adam
95.13 +0.53
99.
0.00
76,49
72.33+t0.44
94,47
98,36
90.37+0.2
74.92+t04
SWATS
95.5
99.17
79.76=2.1
75.17+021
95.12=021
98,38
90,34
74.80
Padam
90.10=
99.46-002
85.64
81.58+03s
95.72=042
98.38
87.07+0.2
80.93
AdaBound
95.02.a.1
99.25_m.m
84,482042
69.27.158
95.42_mm1
81 81.222ns 28.30
79.01
ours
96.81 20.2
99.51 =..01
88.23
84.14
96.12_e.m
99.01 20000
21.36.
81.18
SGI
Adam
SWATS
AdaBound
ours
200
99
sa/00 training loss
training loss
100
84
20
81
uaps (%6)
"a110"
(%)
69
71
2,5
50
25
50
0
50
100
training epoch
training epoch
training epoch
training epoch
Figure 13: The training loss and test accuracy line graph.
3 pretrained models (GPT-2, QWen2-0.5B, and (LaMA-3.2--BB and 5 GLUE datasets (mrpe, rte, qqp. quli, and mnli). All models are fine-tuned using LoRA with rank set to 512 for epochs. Since Adam is the most commonly used and stable optimizer in NLP fine-tuning, Table reports comparisons primarily against Adam. As shown in Table g DOIT consistently achieves higher test accuracy across all tasks and exhibits faster convergence. In terms of peak memory usage, although some differences remain due to memory-relcase behaviors, the overall additional overhead stays below 1,, which aligns with our analysis in Appendix
Table 7: The testAcc. and time for the vanilla method and our DOIT under the head fine-tuning
task
usps
mnist
stl10
model
method
testAcc.
(%)
time (sec.)
testAcc.
(%)
time (sec.)
testAcc. (%)
time (sec.)
SGDM
68,8610m
534
71.59
4443
91.87.1.2
18551
ResNer18
Adam
66.04
543
71.16=146
4876
91.02+1..7
14909
SWATS
68.55
453
73.27
4651
92.02+1.8
14285
Padam
6..3
505
70.41
4743
92.36220.6
19919
AdaBound
67.80
568
72.661031
4944
92.23 1.37
34858
ours
70.641 0,90
531
14.79.3338
4807
93.56 10.41
13265
SGDM
72.27 1o012
4443
77.54
7437
96.6610.33 10.32
78232
ResNet152
Adam
1..73
4876
77.64 L172
7636
96. 10.32
48541
SWATS
71.82.10.7
5121
78.47 10.1
9211
96.66 10.51
115560
Padam
72.3 1 2:0.0
5594
76.18210
8878
96.51 06.51+00 20.30
107856
AdaBound
1..94
4508
78.33
12934
95.44
152592
ours
74,361041
5007
79.60 1060
7645
97.60 10.28
44449
SGDM
91.28+0.30
18551
93.74=a77
50294
92.14 14:771
11464
MobileNet
Adam
89.8640.
12237
93.82+a11
55024
91.3 1
15208
SWATS
90.68
17921
93.90
57655
92.77 10.27
12971
Padam
10.6910.7
18940
92.71 L022
51656
91.6910.29
12941
AdaBound
91.36
17328
93.67
46955
91.39
24367
ours
92.3222..9
13248
94.65
56132
93.28
15042
19

---

Published as a conference paper at ICLR 2026
Table 8: Test accuracy (%), convergence time (sec.), and peak memory (GB) for Adam and our DOIT across more NLP tasks with the LoRA fine-tuning.
model
metric
method
mppe
rtc
qnli
mnli
Adam
71.43
58.35
86.02
82.78
78.08
acc. (%)
ours
72.23
59.13
87.48
83.01
79.14
Adam
62
81
2721
1245
5104
GPT2
time (sec.)
ours
60
81
2681
1203
5049
Adam
6.2
14.9
7.7
7.8
7.7
mem. (GB)
ours
6.4
15.8
8.0
8.0
8.1
Adam
80.93
75.45
89.1
81.55
78.1
acc. (%)
ours
82.84
77.90
89.81
83.25
80.43
Adam
290
560
2727
4652
15412
QWen2-0.5B
time (sec.)
ours
267
503
2701
4439
14972
Adam
17.7
35.2
18.7
10.5
10.5
mem. (GB)
ours
18.4
36.4
19.3
11.3
10.9
Adam
83.71
82.69
89.79
90.24
86.32
acc. (%
ours
85.28
83.37
90.74
90.88
88.13
Adam
335
1222
18677
4342
14743
LLaMA-3.2-IB
time (sec.)
ours
321
1193
18593
4278
14631
Adam
22.4
45.8
25.5
25.5
25.5
mem. (GB)
ours
23.7
47.8
26.3
27.0
26.6
Friedman and Nemenyi Tests. To compare the performance different methods, we conducted statistical tests the accuracy results of 15 experiments involving methods listed in Table 2 and Table 6 In the Friedman Test, we assume that all methods have the same average rank. Let R, denote the average rank of the j-th method, number of datasets. The chi-square can then be calculated as:
as the number of methods, and N 15 as the
12N
k(k + 1)2
X
R
63.4.
k(K
4
j=1
and the FF value is,
(N - 1)
Fp N(k - 1) X3
~ 15.7.
At the significance level of 0.05, the crit-
ical value Fa Fo.05(8, 14) ~ 2.699. Since Fr F., we reject the hypothesis, indicat- ing that there are sigmificant differences among the methods. Next, we further perform the Ne- menyi test and compute the critical distance as:
SGD
SGDM
Adagrad
RMSprop
Adam
SWATS
Padam
AdaBound
k + 1) = 3.102,
ours
CD = qa
6.V
6
where = 3.2 when k and a 0.05. The visualization of the Nemenyi test, as shown in Figure 14 indicates that DOIT performs sig- nificantly better than all other methods except SGDM and Padam. Moreover, although the differ- ences between DOIT and SGDM/Padam are slightly below the critical distance, DOIT consistently outperforms both methods across multiple datasets. Therefore, we believe that DOIT still holds an overall performance advantage.
rank
Figure 14: Visualized Nemenyi test result.
F
MORE DETAILS AND RESULTS ABOUT INDEPENDENT EXPERIMENTS
Due to space limitations, we only provided summary of the independent experiments in Sec- tion 5.3 Here, we present additional details and experimental results.
20

---

Published as a conference paper at ICLR 2026
F.1 VARYING SEARCH SPACE
To investigate the performance of DOIT under a varying optimizer search space, we conducted ex- periments along two dimensions: the number of optimizer types and the number of hyperparameter types. (i) In the optimizer-type experiment, we gradually expanded the search space starting from SGDM by adding SGD, Adagrad, RMSprop, ASGD, Adam, AdamW, NAdam, and Adamax. For each optimizer, the learning rate selected from [0. (0.1,0.01,0.001) while other hyperparameters were kept at their default values. (ii) As for the hyperparameter experiment, we systematically ex- panded the original HP space described in Section [5.1 (where learning rate [0.1, 0.01, 0.001 by including the following components: (1) weight decay for SGD, (2) momentum for SGDM, (3) weight decay for Adagrad, (4) weight decay for RMSprop. (5) alpha for RMSprop, and (6) weight decay for Adam. The ranges for these additional hyperparameters are as follows: weight decay values [le-2, 1e-3, lc-4], momentum [0.5, 0.6, 0.7, 0.8, 0.9]. and alpha E [0.5, 0.6, 0.7, 0.8, 0.9].
Figure 12 illustrates the performance of DOIT compared to the baseline methods as the search space diversifies. For the baseline methods, we report the best accuracy achieved within the search space. Overall, DOIT demonstrates robust performance and consistently surpasses the baseline methods, even as the number of candidate hyperparameters increases. Furthermore, the expanded search space does not noticeably increase DOIT's time cost. As discussed in Section [42 and the Appendix [B] the change in search space only impacts the surrogate model's prediction time within the acquisition function calculation. This overhead is minimal relative to the total training time amounting to just 0.15% even when evaluating 10 configurations per search.
In addition, we conducted further experiments using Optuna-based hyperparameter optimization on ResNet18, tuning both optimizer-specffic parameters (i.e.,. learning ratee [le 5, lc 1], momentume [0.5, 0.99), weight decayE [Le - 6, le - 2]. and 31 E [0.8, 0.999], B2 E [0.9. 0.999 for Adam) and scheduler-related settings (i.e., scheduler typeε (CoelnAnnel, StepLR), warm-up stepse [0, In Repocck) number of epochs [50. 100)). As for DOIT, we search init step Vini E [10, 100] and switch step size r [Le 6, 1e 2]. As shown in Table even after extensive hyper- parameter tuning, DOIT consistently outperforms well-optimized single optimizers across datasets, demonstrating its robustness and the advantage of dynamic optimizer switching.
Table 9: Performance comparison under Optuna-based hyperparameter optimization on ResNet18. 8. Results show test accuracy (%) on usps and st//0. The column "default" denotes accuracy without BO tuning, while the remaining columns report results with varying search depths (number of trials). DOIT consistently outperforms the best-tuned single optimizers.
Dataset
Method
default
5
20
25
SGDM
96.10
95.22
95.86
95.86
95.86
96.21
usps
Adam
95.13
95.86
96.16
96.30
96.51
96.51
DOIT
96.81
96.16
96.91
96.91
97.12
97.32
SGDM
86.99
87.18
87.83
87.88
87.88
88.13
10
Adam
76.49
86.81
86.73
87.87
87.88
88.03
DOIT
88.23
88.18
88.19
88.49
88.83
88.83
F.2 ABLATION EXPERIMENTS
We conducted a series of ablation studies to investigate the effectiveness of key design components.
The selection of optimizer selection strategy and design of acquisition function. Compared to switching optimizers with simple strategies, DOIT employs the transferability assessment wt and variance halving in its acquisition function. To estimate the effectiveness of our design, we com- pared DOIT with two simple strategies (random switch "random" and periodic replacement "cycli- cal") and two ablation versions (without transferability assessment "w/o wt" and without variance halving "woo halve"). The experimental result in Table I0 shows that simple random periodic switching fails to produce high test accuracy. In addition, the usage of transferability assessment and variance halving both effectively enhances the adaptability of the current task, resulting in im-
21

---

Published as a conference paper at ICLR 2026
proved accuracy (up to 2%) and lower variance. Additionally, also compared DOIT with these strategies under other models (i.e., ResNet152 "RN152" and MobileNet V2 "MN"), as shown in Table I and Table [2
Table 10: Experimental results for different switch strategies under full training on ResNet18.
method
usps
mnist testAcc, (%)
trainAce, (%)
testAcc. (%)
time (sec.)
trainAce. (%)
time (sec.)
random
98.3622.2
94.27
225
98.91
98.331046
1916
cyclical
97.3821m
92.88 1M
223
99.49220M
28
1914
w/o
99.55
95,82
205
99.72+*09
99.25
+0.07
2101
w/o halve
29.54.0m
95.70 95
226
99.67
99.19 200s
2095
ours
99.67.2.19
96.51
227
99.85 10.17
99.48
1965
Table 11: The independent experimental result for transferability assessment.
Table 12: The independent experimental result for different optimizer switching strategies.
metric
w/o w/
with wi
metric
random
cyclical
MN RN152
trainAce. (%)
80.47
81.10
MN RNI52
trainAcc. (%)
76.56
77.79
81.10
testAcc. (%)
72.6
75.29
testAcc. (%)
71.49
71.95
75.29
time (sec.)
3531
3347+9
time (sec.)
3271
3299
3347+9
trainAcc, (%)
98.38
98.42
trainAce, (%)
97.58
98.01
98.42
testAcc. (%)
91.57
91.93
testAcc. (%)
90.33
90.73
91.93
time (sec.)
26084
23271+8
time (sec.)
24997
23712
23271+8
The initial selection method. We compared the impact of using random selection and weighted selection during the initial phase on subsequent training. In the experiments, 20 epochs and used a switch step size of T 25. Compared with
random
ours
set the initial phase
100
validation accuracy (%)
random selection, the weighted selection in DOIT significantly en-
hances the stability of the surrogate model, which reduces variabil-
usps
ity in the training outcomes, as shown Figure
The model compression technique. Table 13 illustrates the effects of different feature compression techniques on training results. For the selected tasks (i.e., usps and mnist), simple methods such as random projection (RP) and PCA outperform the more complex UMAP, suggesting that basic compression techniques are sufficient for surrogate modeling. This is consistent with our design rationale: PCA provides stable, continuous dindenensional projections that align well with surrogate models like GPs, while methods such as UMAP or t-SNE tend to ineur higher computational costs and produce embeddings less suitable for iterative training. Together, these findings indicate that lightweight compression techniques not only reduce computational over- head but also deliver reliable performance, making them practical choices in the DOIT framework.
init
0 training epoch
100
Figure 15: Ablation results of initial methods.
Table 13: Experimental results for different compression techniques under full training on ResNet18. 8.
method
usps time (sec.)
mnist test/cc. (%)
trainAcc. (%)
testAcc. (%)
trainAcc. (%)
time (sec.)
RP
99.72:0.12 98.562145
96.44ian
201
99.61
99.2210.1
1869
UMAP
95.15
253
99. 99.734400
98. 98.9 20.3
2182
PCA
99.58
96.34
222
99.86
99.48
1974
22

---

Published as a conference paper at ICLR 2026
F.3 HYPERPARAMETER ANALYSIS
We performed an experimental analysis on the settings of hyperparameters in DOIT, including the initial step Pini, switch step size 7, and the number of PCA components. In the experiments, the default hyperparameter configuration to Pins = 20, 7 25, and n_components= 2. Based on the results presented in Figure 16
we have made the following observations.
init step Rimt Thanks to the ongoing updates of the surrogate model during training in DOIT, even a small initial step (i.e., Rani = 20) can produce models with high test accuracy.
switch step size т: A smaller step size facilitates quicker switching of the optimizer, which en- hances accuracy (e.g.,
20% of an epoch). Conversely, a larger step size makes it more challenging to collect training data for the surrogate model, resulting longer switching cycle and greater variance in the results.
Selecting a small number of PCA components (for example, 2) can often yield good performance in DOIT. On the other hand, using larger number of components may impair the surrogate model's ability to learn effectively, resulting in greater variance in test accuracy.
PCA components number:
sub-selected layers for compression: We used only the last layer input to the surrogate model previous experiments. To further analyse the influence of selected layers, we constructed an independent experiment for the selected layer, shown in Table 14 As shown in the table, adding more layers yields only marginal accuracy gains but incurs substantially higher time costs, and using many layers may even degrade performance due increased difficulty in surrogate modeling.
Variance weight a: We analyzed the sensitivity of the acquisition function variance weight, shown in Table [5] On mnist, performance is stable across all tested values. On usps, accuracy is more sensitive but remains stable with moderate À (0.3-0.5).
Halving period We also varied the halving period that controls how frequently the variance weight is reduced. Results in Table 16 show that larger halving periods (c.g., 1/2 or 1/4 /4 of training) yield more stable performance, especially on usps, while mnist remains robust across settings.
99
96.5
test accuracy (%)
95
test accuracy
95.5
test accuracy
95
usps
91
usps
asps
99.5
99.7
99.5
99.2
minss
99.0
mnist
99.3
miist
10
100
10%
100%
10
(a) init step Niii
(b) switch step epoch
PCA components
Figure 16: The experimental results of hyperparameter analysis.
Table 14: Sensitivity to the sub-selected layers for compression.
only cls.
+1 layer
+2 layers
+3 layers
test acc. (%)
95.02
95.12 246
95.07 727
95.07
time (sec.)
243
820
G DISCUSSION ON STATE INHERITANCE
In framework. DOIT resets optimizer states upon switching rather than inheriting them. This design choice is deliberate and generally does not cause noticeable instability. Below we provide both theoretical and empirical justification for this decision.
23

---

Published as a conference paper at ICLR 2026
Table 15: Sensitivity to the variance weight a in the acquisition function (higher better).
0.1
0.3
0.5
0.7
0.9
usps muist
95.02
95.57
95.72
92.33
96.21
99.06
98.81
98.78
98.81
99.00
Table 16: Sensitivity the halving period (fraction of training after which the variance weight is halved).
Repoch
3 Repoch
Repoch
3 nepooh
usps
89.64
95.22
94.92
93,42
95.08
mnist
99.22
99.20
99.21
99.20
98.62
Alignment with the core hypothesis. DOIT selects the optimizer best suited to the current param- eter state. Carrying over internal states such as momentum or running averages (e.g.. transferring from Adam SGDM) may misalign with the dynamics of the new optimizer, undermining the purpose of switching.
Empirical evidence. We compared DOIT with and without state inheritance across multiple datasets (e.g., transferring momentum between Adam and SGDM), As shown in Table 17 simple inheritance strategies do not consistently improve performance and may even lead to degradation instability.
Technical complexity. Robust state transfer across optimizers with different internal dynamics (e.g., momentum in SGD vs. second-order statistics in Adam) remains an open problem. De- signing such mechanisms goes beyond the scope of this paper, but we consider it important direction for future work.
Training dynamics of DOIT. analyzed Sec. 5.2 and visualized in Figs. any instability due to state resetting primarily occurs in the early phase of training, when the surrogate model still adapting. Later, optimizer selection stabilizes and no significant loss spikes are observed. In fact, resetting states sometimes mitigate instability by preventing overly aggressive updates (e.g., momentum inheritance leading to divergence).
and 13
Table 17: Comparison of DOIT with and without state inheritance.
Method
usps
mnist
stl
State inherit
95.02
99.38
77.13
State reset
96.11
99.15
88.11
H LLM USAGE STATEMENT
Large language models (LLMs) were only used in this work to assist with writing, including lan- guage polishing and grammar checking.
24


Reviews:
[{"review_id": "GezHWfXfPm", "reviewer": "Reviewer_PbZG", "summary": "This paper introduces dynamic interleaving of optimizers (DOIT) framework that adaptively switches among multiple optimizers during training based on the training state. The method constructs and continually updates Gaussian surrogate models for each optimizer to estimate their expected performance under different conditions. At each switching step, the optimizer predicted to yield the highest improvement is selected for the next training phase.", "questions": "Some questions on hyperparameters and heuristics: \n- how long is the experience collection phase per dataset and what % of training is dedicated to it? is it a tunable parameter?\n- what are hyperparameters of gaussian processes model? how are they chosen for each dataset/task?\n- how is switch cycle tau chosen? Is it an extra tuning parameter?\n- which layers are sub-selected for surrogate model for each dataset? \n- for a given selected optimizer in a switch cycle based on acquisition function score, how are optimizer hyper-parameters chosen (such as learning rate, momentum, weight decay, betas if applicate etc)?\n\nMoreover, can authors please elaborate on the key runtime/memory bottlenecks in the current approach? What are the most and least expensive parts in the approach and how do they change (if they do) based on DOIT hyperparameters?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Novel approach for interleaving optimizers during training based on surrogate quantitative performance\n- Experiments are conducted on a variety of datasets (6 CV, 2 NLP and 3 additional tasks) with two settings: full training and fine-tuning setups and compared against standard optimizer baselines (although a bunch of these tasks are small as per today's standards, see weaknesses below)", "weaknesses": "- The approach introduces a lot of additional hyper-parameters (see questions below). Moreover, optimal DOIT hyperparameters are different per dataset which limits the pratical utility of the approach and introduces additional tuning overhead for the practitioner. \n- The proposed approach incurs high memory and runtime overhead which are reduced by arbitrary heuristics such as PCA over parameters, additional sub-selection of layers (only last). It's unclear if these heuristics will hold for large-scale ML workloads. Similarly, acquisition function is hand-designed with heuristics. These are central components of the proposed approach and non-trivial changes to the approach may be required to make it scalable to standard ML tasks on which optimizers are usually benchmarked (GPT-2, ViT, ResNet50, AlgoPerf benchmark, etc).\n- Experiment datasets are small as per today's standard (4/6 CV datasets are mnist-scale, imagenet is the only relevant one) and also heavily biased towards vision tasks (6 CV datasets, coco object detection)\n- The approach requires extra surrogate model (Gaussian processes) which greatly limits scalability of the proposed approach (n^3 complexity). \n- In order to collect experience data for surrogate model, training is performed with random optimizer configurations -- wouldn't this lead to irreversible damage to the parameters of the model which may be hard to recover from at later stages in the training?\n- While DOIT switching is mainly guided by the short-term benefits of each optimizer, practitioners aiming for highly efficient and performant training also consider long-term effects, taking into account the total number of iterations, final accuracy, and selecting hyperparameters such as learning rate, weight decay, and their schedules accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DJoiWGH9ek", "reviewer": "Reviewer_5cJs", "summary": "This paper introduces a novel method called Dynamic Optimizer Interleaving Training (DOIT), which dynamically switches between optimizers during training to enhance model performance. Traditional training processes typically rely on a single static optimizer or a simple hybrid, missing opportunities to adapt to changing dynamics during training. DOIT builds surrogate models to predict optimizer performance based on the current parameter states and selects the most suitable optimizer using an acquisition function. Experiments across tasks such as classification, translation, and object detection show that DOIT improves convergence speed (2% to 10%) and accuracy (1% to 3%). Further case studies validate its effectiveness.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a dynamic optimizer to improve training performance.\n\n2. The paper provides comprehensive experiments in both the vision and NLP domains, validating the proposed method's effectiveness.\n\n3. The paper includes a comparison of training efficiency, which is quite useful.", "weaknesses": "1. The method requires retaining multiple optimizers, which may lead to high memory usage. The authors should include a direct comparison to address this concern.\n\n2. As mentioned by the authors, the method does not support optimizers with momentum. Since momentum is a key mechanism in current mainstream optimizers, I recommend that the authors propose a feasible solution to extend their method to support momentum-based optimizers.\n\n3. While some details still need clarification, I understand that the method retains multiple optimizers and uses strategies like performance-aware weighting after a certain number of steps (Eq. 6) for weight updates. In this process, where exactly is the \"training dynamic\" reflected? Additionally, doesn’t this approach resemble ensemble learning? Furthermore, the method seems related to meta-learning; it would be helpful for the authors to analyze and compare these connections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TZ2VRxfzFE", "reviewer": "Reviewer_MSZS", "summary": "This paper proposed a new framework called Dynamic Optimizer Interleaving Training (DOIT) for training deep neural networks. The main idea is to use a surrogate model, more specifically, a Gaussian Process (GP), to assess the performance of the optimizers, and then dynamically select the best one from the candidates. Experiments on several datasets have been conducted to demonstrate the effectiveness of the proposed method.", "questions": "First, the authors should address the issues mentioned in the \"Weaknesses\". Despite these weaknesses, several more things should be clarified:\n\n1. The authors only report the FLOPs of the proposed method, but it is also important to consider the memory overhead of the additional operations, such as PCA compression and GP prediction.\n\n2. It seems that the optimizer configuration candidates lie in a discrete space, and thus, it could be tricky to construct the candidates.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow.\n2. The performance of the proposed method seems good.", "weaknesses": "1. The novelty of this work is not significant enough. Specifically, the key component of the proposed framework is a surrogate model constructed by GP, which, however, is not unusual and has been widely used in hyperparameter optimization. Despite this key component, other designs are mostly engineering and heuristic, such as the performance score and the acquisition function.\n\n2. There is no theoretical convergence guarantee for the proposed method. Since the training process by the proposed framework includes the switch of multiple optimizers and corresponding hyperparameters, the convergence behavior could be more complex than a single optimizer, and thus, a theoretical convergence guarantee can be useful to guide practice. Besides, it would also be interesting to analyze how good the surrogate function is and how it could affect the final performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k4CqvTs3jA", "reviewer": "Reviewer_ffH8", "summary": "This paper introduces DOIT (Dynamic Optimizer Interleaving Training), a novel method that dynamically switches optimizers during training. The core hypothesis is that different optimizers are better suited for different training states. DOIT models this problem by building Gaussian Process (GP) surrogate models to predict the \"short-term\" performance of different optimizers based on a PCA-compressed representation of the current parameter state. A well-designed acquisition function—which considers predicted performance, uncertainty, task transferability, and training progression—is used to select the optimal optimizer for the next training cycle.\n\nThe authors provide extensive empirical validation across a wide range of tasks (image classification, text classification, machine translation, object detection), models (ResNet, ViT, RoBerta), and settings (full training, PEFT). The results consistently show that DOIT achieves higher accuracy (1-3% gains) and faster convergence (2-10% faster) compared to a large suite of static and simple-hybrid optimizers.", "questions": "1.Some pervious work also explore the potential of dynamic hybridization of different optimizer, such as HUB (hybrid-update-based optimization strategy). They emphasized the hybridization with learned optimizers. Could you testify if DOIT could also handle the switching timing for those optimizers?     \n\n2. How was the subset of layers for PCA compression chosen for the main experiments (e.g., for ViT and RoBerta)? How sensitive is DOIT's performance to this choice?\n\n3.The acquisition function includes a transferability weight $\\omega_t$. Does this imply the surrogate models must be re-trained from scratch for every new task, or can the GP models themselves be transferred?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.The motivation of this paper is direct and clear. The paper presents a principled approach to optimizer selection. Moving from static selection or simple two-stage hybrids (like SWATS) to dynamic, state-aware selection framework is a significant conceptual leap.\n\n2.The breadth of tasks, datasets, and models is comprehensive and convincing. The inclusion of ablation studies, cost analysis, and case studies on switching behavior thoroughly validates the method's design and effectiveness.\n\n3.Initially i questioned if the overhead of training multiple GP online would be significant. However, the authors provide a detailed cost analysis and empirical measurements suggesting that the overhead from surrogate modeling and selection is minimal (often <1%), making the method practical.", "weaknesses": "1.The most significant weakness is that the method resets optimizer states (e.g., momentum, second-moment estimates) upon every switch. This seems counter-intuitive and potentially disruptive, as the optimizer loses all accumulated history. The authors defend this in Appendix, arguing that state inheritance is complex and their empirical tests (Table 15) showed it can destabilize training. However, this should be further proven. \n\n2.The surrogate model's input is a PCA projection of some model layers. The paper is not perfectly clear on which layers are chosen for the different models, and how sensitive the performance is to this choice (e.g., classifier only vs. final N blocks).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)