Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How
Authors: Sebastian Pineda Arango
Fabio Ferreira
Arlind Kadra
Frank Hutter
Josif Grabocka
Venue: ICLR 2024 oral (2024)
Keywords: Finetuning
pretrained model hubs
transfer learning
hyperparameter optimization
meta-learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We learn to jointly and efficiently select pretrained models to finetune and their hyperparameters.

Abstract: With the ever-increasing number of pretrained models, machine learning practitioners are continuously faced with which pretrained model to use, and how to finetune it for a new dataset. In this paper, we propose a methodology that jointly searches for the optimal pretrained model and the hyperparameters for finetuning it. Our method transfers knowledge about the performance of many pretrained models with multiple hyperparameter configurations on a series of datasets. To this aim, we evaluated over 20k hyperparameter configurations for finetuning 24 pretrained image classification models on 87 datasets to generate a large-scale meta-dataset. We meta-learn a gray-box performance predictor on the learning curves of this meta-dataset and use it for fast hyperparameter optimization on new datasets. We empirically demonstrate that our resulting approach can quickly select an accurate pretrained model for a new dataset together with its optimal hyperparameters.

Published as a conference paper at ICLR 2024
QUICK-TUNE: QUICKLY LEARNING WHICH PRE- TRAINED MODEL TO FINETUNE AND HOW
Sebastian Pineda Arango, Fabio Ferreira, Arlind Kadra, Frank Hutter & Josif Grabocka Department of Computer Science University of Freiburg pinedaêcs uni uni-free uni-freiburg.co
ABSTRACT
With the ever-increasing number pretrained models, machine learning practi- tioners are continuously faced with the decision of which pretrained model to use, and how to finetune it for a new dataset. In this paper, we propose methodology that jointly searches for the optimal pretrained model and the hyperparameters for finetuning it. Our method transfers knowledge about the performance of many pre- trained models with multiple hyperparameter configurations To this aim, we evaluated over 20k hyperparameter configurations for finetuning 24 pretrained image classification models on 87 datasets generate large-scale meta-dataset. We meta-learn a gray-box performance predictor on the learning curves of this meta-dataset and use for fast hyperparameter optimization on new datasets. We empirically demonstrate that our resulting approach can quickly select an accurate pretrained model for new dataset together with its optimal hyperparameters. To facilitate reproducibility, we open-source our code and release our meta-dataset
series of datasets.
INTRODUCTION
Transfer learning has been game-changer in the machine learning community, finetuning pre- trained deep models on new task often requires much fewer data instances and less optimization time than training from scratch (Liu al. 2021 You et al. 2020). Researchers and practitioners constantly releasing pretrained models of different scales and types, making them accessible to the
public through model hubs (a.k.a. model zoos model portfolios) (Schürholt al. [022 Ramesh & Chaudhari 2022). This raises a new challenge, as practitioners must select which pretrained model to use and how to set its hyperparameters You al. 2021b), but doing so via trial-and-error is time-consuming and suboptimal.
In this paper, we address the resulting problem of quickly identifying the optimal pretrained model for new dataset and its optimal finetuning hyperparameters. Concretely, we present Quick-Tune Tune, Combined Algorithm Selection and Hyperparameter Optimization (CASH) (Thornton et al.) 2013) technique for finetuning, which jointly searches for the optimal model and its hyperparameters in Bayesian optimization setup. Our technical novelty is based on three primary pillars: i) gray-box hyperpurameter optimization (HPO) for exploring learning curves partially by few epochs and effectively investing more time into the most promising ones, ii) meta-learning for transferring the information of previous evaluations related tasks, and iii) cost-awareness for trading off time and performance when exploring the search space. By utilizing these three pillars, our approach efficiently uncover top-performing Deep Learning pipelines (i.e., combinations of model and hyperparameters)
In summary, we make the following contributions:
We present an effective methodology for quickly selecting models from hubs and jointly tuning their hyperparameters.
https://github.ce ott.:////o..eaaiirerrroorro/oooo

---

Published as a conference paper at ICLR 2024
We design extensive search space that covers common finetuning strategies. In this space, we train and evaluate 20k model and dataset combinations to arrive at a large meta dataset in order to meta-learn gray- performance predictor and benchmark our approach.
We compare against multiple baselines, such as common finetuning strategies and state-of- the-art HPO methods, and show the efficacy of our approach by outperforming all of the competitor baselines.
RELATED WORK
Finetuning resumes the training on a new task from the pretrained weights. Even if the architecture is fixed, the user still needs to specify various details, such as learning rate and weight decay, because they are sensitive to the difference between the downstream and upstream tasks, or distribution shifts (Li et al., 2020 Lee et al., 2022). A common choice is to finetune only the top layers can improve performance, especially when the data is scarce (Yosinski et al., 2014). Nevertheless, recent work proposes to finetune the last layer only for some epochs and subsequently unfreeze the rest of the network (Chen et al., 2019a] Wang et al., 2023), to avoid the distortion of the pretrained information. To reduce overfitting, some techniques introduce different types of regularization that operate activation
Finetuning Strategies
(Kou et al.) 2020 Li et al., 2020 Chen et al.) (01 parameter wise (Li et al. 2018), or directly using data from the upstream task while finetuning (You et al., 2020 Zhong et al., 2020). No previous work studies the problem of jointly selecting the model to finetune and its optimal hyperparameters. Moreover, there exists no consensus on what is the best strategy to use or whether many strategies should be considered jointly as part of a search space.
Model Hubs It has been common practice in the ML community to make large sets of pretrained models publicly available. They are often referred to as model hubs, zoos, or portfolios. In computer vision, in the advent of the success of large language models, more recent trend is release all- purpose models (Oquab et al. 2023) Radford et al. 2021 Kirillov et al. (023) which aim to perform well in a broad range of computer vision tasks. Previous work has argued that a large pretrained model can be sufficient for many tasks and may only need little hyperparameter tuning (Kolesnikov) al. 2020). However, recent studies also show strong evidence that scaling the model size does not lead to molelodelli solution in computer vision (Abnar et al.) (022). Besides presenting more diversity and flexible model sizes for adapting to variable tasks and hardware, model hubs can be used for regularized finetuning YYou et al. 2021a). learning hyper-networks for generating the weights (Schürholt et al. (022). learning to ensemble different architectures (Shu et al. 2022).
ensembling the weights of similar architectures (Wortsman al. [022b; Shu et [2.2] Wortsman
let al., 2022a), or selecting a suitable model from the pool (Cui et al. 2018 Bao er al., 2019; Tran
al. 2019a Nguyen et al.) 2020 You et al. (202Tb] [olly al. 202 Previous work using model
hubs does not analyze the interactions between the used model(s) and the hyperparameters and how
to set them eficiently.
HPO, Transfer HPO, and Zero-Shot HPO Several methods for Hyperparameter Optimization
(HPO) have been proposed ranging from simple random search (Bergstra & Bengio, 2012a) to fitting surrogate models of true response, such as Gaussian processes (Rasmussen & Williams
2006), random forests (Hutter et al. 2011). neural networks (Springenberg et al. (016)). hybrid techniques (Snoek et al. 2015), and selecting configurations that optimize predefined acquisition functions (Wilson al. 2018). There also exist multi-fidelity methods that further reduce the wall-clock time necessary to arrive at optimal configurations (Li et al. 2017 Falkner et al., 2018a Awad et al. [2011a] Shala et al., 2223a Kadra et al. 2023). Transfer HPO can leverage knowledge from previous experiments to yield strong surrogate model with few observations on the target dataset (Wistuba Grabocka 2021a Pineda Arango & Grabocka 2023, Shala et al., 2023b Salinas et al.) 2020). Methods that use meta-features, I.e., dataset characteristics that can be either engi- neered (Feurer et al., 2015 Wistuba et al.) (0166) or learned (Jomaa et al., 202T). have also been proposed to warm-start HPO. Zero-shot HPO has emerged an efficient approach that does not require any observations of the response on the target dataset, e.g. approaches that are model free and use the average performance of hyperparameter configurations over datasets (Wistuba et al. (2015) or approaches that meta- learn-leaann surrogate models with a ranking loss (Khazi et al.) [023)] Winkelmolen et al. 2020: Öztürk et al. 2022). In contrast to previous work, we propose to not only use the final
2

---

Published as a conference paper at ICLR 2024
performance of configurations but to learn Gaussian Process-based to predict the performance of partial learning curves as formulated by gray-box HPO approaches (Hutter et al. 2019).
MOTIVATION
Before introducing our method, we want remind the reader about the importance of searching for the optimal pretrained neural network from pool of models. Our main premise is that there is silver bullet model that fits all the finetuning tasks. To illustrate this fact, we computed the error rates of a group of 24 efficient models (detailed in Section ( from the timm library (Wightman] 2019) on all the 26 datasets of the Extended split of MetaAlbum (Ullah al. 2022) (details in Section (5). For every model, we use its best per-dataset hyperparameter configuration found by comprehensive HPO. Higure I shows the ranks of the 24 models for the 26 datasets, demonstrating that there is very little regularity. In particular, there exists no single model that ranks optimally on all datasets, even if we optimize its hyperparameters for each dataset. Since there exists silver bullet model, and considering that there is large number of pretrained models available in recent hubs, then how can quickly select the best model for dataset?
5
Model index
10
15
10
20
Dataset index
10
20
Figure 1: Ranks of model perfor- mances across datasets.
4 QUICK-TUNE: COST-EFFICIENT FINETUNING
Following our motivation, we aim to find the best pipeline = {m, 1}. X' within a search space X' == M A comprising a model hub m E M and set of hyperparameters À E In this section, we detail how we solve this problem efficiently in order to yield competitive anytime performance.
4.1 QUICK-TURE TUNE
We follow an efficient Bayesian Optimization strategy to search for the optimal pipelines, a similar style to recent state-of-the-art approaches in HPO (Wistuba & Grabocka 2021b Wistuba al. 2022). At every iteration, our method Quick- Tune fits estimators that predict the performance of pipelines and their cost (for details, see Section (.2). Then it uses acquisition function (detailed in Bection[4.3) to select the next pipeline to continue finetuning for an incremental number of epochs. Finally, our method evaluates the loss and the runtime cost and adds it to the history. This procedure is repeated until a time budget is reached. We formalize these steps in Algorithm [] where we use the validation loss as performance metric. The entire procedure is sped up by starting from meta-learned surrogate as described in Section 4.4
Algorithm 1: Quick-Tume Algorithm
Input: Search space of pipelines E X' Epoch step At Pipeline with the smallest observed loss Select randomly a pipeline x' E X and evaluate it for At epochs; Initialize the history while budget do
Output:
((11, Al, Al), c(x', At))
Update the performance predictor from H. using Equation
Update the cost estimator è from H using Equation
Select the next pipeline x² using Equation
)) and measure the cost c "("") Update the history H H U (x - ((x)) l (x r(x")) ((x)))))))))))
Evaluate the performance e (x", [(x")
end
10 return arg min (f(x,t) (x, t, E(1 (). H}:

---

Published as a conference paper at ICLR 2024
4.2 PERFORMANCE AND COST ESTIMATORS
Learning curves record the performance of Deep Learning pipelines at different time steps, such as the validation loss versus the number of epochs. The performance denoted as (z, t), and the runtime cost for training the pipeline z until step t is e (z, t). The history of all observed learning curves for pipelines denoted as H. {(z,, f, (z,, t,) (Z t,)))
the pipeline x at step is
Our method learns a probabilistic performance estimator (a.k.a. surrogate) defined as & (æ, 1; and parameterized with 0. We train the surrogate to estimate the true performance & from H as:
0" := arg min ((r...(...)))) [- logg (z, 8
Concretely, the surrogate i implemented as a deep-kernel Gaussian Process regressor (Wistuba & Grabockal 2021a). In addition, train cost estimator ((),t;y) in the form of Multilayer Perceptron with parameters to predict the ground truth costs as:
min ((s.tso(2.00)) (c(xr.) -c(x
4.3 COST-SENSITIVE ACQUISITION FUNCTION
We propose a cost-sensitive variant of the Expected Improvement (Jones et al. 1998) (EI) acquisition to select the next pipeline to evaluate within Bayesian Optimization framework, defined as:
EI H. E(xx, ((x)) à ((,,T(z)) -c 10.0x,7(00 -11)
arg E ((x max (erin ixx, ((r,T(2)),,0
x² arg max ZEX
#EA
à ((1, -C x. T Dt)
The numerator of Equation 3 introduces mechanism that selects the pipeline that has the largest likelihood to improve the lowest observed validation еттог at the next unobserved epoch r(x) of pipeline æ. The denominator balances out the cost of actually finetuning pipeline æ for At epochs. H) + At, where At denotes the number of epochs to finetune from the last observed epoch in the history. the pipeline is not in the history, the query epoch is 7xx) At. Simply put, if the validation loss of is evaluated after every training epoch/step (At 1) and has been evaluated for & epochs/steps, then r(x) = 1. As result, select the configuration with the highest chance of improving the best-measured loss at the next epoch, while trading off the cost of finetuning it. Concretely, the best observed loss is gmin min (fE(1, 7 (z)) F(z), (z, r(z)),) .-) E H)). If no pipeline has been evaluated until T(x), i.e. (2),)))))))) & H. then fani = min ((e(z, (1) i, e (x, (), E H,, < T(x))).
((r) is defined for pipeline æ as rxx) = max(t'( (x,,'' I', :;
4.4 META-LEARNING THE PERFORMANCE AND COST ESTIMATORS
crucial novelty of our paper is meta-learn BO surrogates from existing pipeline evaluations other datasets. Assume we have access to a set curves for the validation errors and the runtimes e of pipelines over a pool of datasets, for series of N epochs. We call the collection of such quadruple evaluations meta-dataset (M) Usex x UnED ULEDIN] {(x, t, (x, t, d) (x, t, d))}, where explicitly included the dependency of the performance and cost curves the dataset. To contextualize the predictions on the characteristics of each dataset, we use descriptive features d E D to represent each dataset (a.k.. meta-features).
We meta-learn a probabilistic validation error estimator i (xx, d: 0), and a point-estimate cost predictor è (x. I, d; 7) from the meta-dataset H(M) by solving the following objective functions:
4

---

Published as a conference paper at ICLR 2024
g(M) : arg min (cc.
[- -log t(( 2,t,d,((
(4)
(M) arg min (ot. (c(z, (c( (x,,tdid)
(5)
After meta-learning, we use the learned weights to initialize the performance and cost predictors <- ((M) and 7 + (M) before running Algorithm| As result, our method starts with a strong prior for the performance of pipelines and their runtime costs, based the collected history H(M) from evaluations on prior datasets. We provide details about the meta-learning procedure in Algorithm (Appendix A.3)
QUICK-TUNT
5.1
QUICK-TUNE SEARCH SPACE
While our proposed method is agnostic to the ap-
plication domain, the of pretrained models and
Top-1 Accuracy
hyperparameter space to choose from, we need to
instantiate these choices for our experiments, In this
paper, we focus on image classification and base our
study on the timm library [Wightman] 2019), given
its popularity and wide adoption in the community. It contains a large set of hyperparameters and pretrained models ImageNet (more than 700). Concerning the space of potential finetuning hyperparameters, we select subset of optimizers and schedulers that well-known and used by researchers and practition- We also include regularization techniques, such as data augmentation and drop-out, since finetuning typically applied in low data regimes where large architectures easily overfit. Additionally, we mod- ified the framework to include common finetuning strategies, such as methods to select the percentage of layers to finetune YYsiinski et al., 2014).
Non Pareto-optimal
Pareto optimal
No. of Parameters (Millions)
The subset of Pareto optimal pre- trained models with respect to the predictive accuracy and model size.
Figure 2:
linear probing [Wang et al.) 2023), stochastic norm (Kou et al. 2020), Co-Tuning You et al. 2020 ,
DELTA (Li et [ 2019), BSS (Chen et 2019b) and SP-regularization (Li et al. 201 8). The last five methods are taken from the transfer learning library (Junguang Jiang & Long 2020). Although consider these well-known and stable finetuning strategies, foresee the widespread adoption of
new approaches such as LoRA (Hu et al.) 2021). They are complementary to our method and can be easily interpreted search space in Table I indicating explicitly the conditional hyperparameters with a "g" For a more detailed description of our search space. including the hyperparameter ranges and dependencies, point the reader to Table of Appendix B As we are interested in time efficiency and accuracy, we select the Pareto optimal models from the large set of ca. 700 pretrained architectures in the rimm library. Specifically, given model m E MTimm with Top-1 ImageNet accuracy fimageNet (m) and ((m) number of parameters, we build our final model hub based on the multi-ohjective optimization among the predictive accuracy and model size by solving Equation 6 Subsequently, we obtain set of Pareto-optimal models as shown in Figure and listed in Table of Appendix B
extension of the pipeline search space. We list all the hyperparameters of our
M
m m E arg max [fmaggNos (m), -5(m)]
(6)
mE
5.2 META-DATASET GENERATION
We created large meta-dataset evaluated learning curves based on the aforementioned search space. Overall, we finetuned the 24 Pareto-optimal pretrained models 86 datasets for different hyperparameter configurations (details in Table 6 Appendix B. For every dataset. we sample hyperparameter configurations and models uniformly at random from the search space of Table
5

---

Published as a conference paper at ICLR 2024
Table 1: Search Space Summary.
Hyperparameter Group
Hyperparameters
Percentage of the Model to Freeze, Layer Decay,
Finetuning Strategies
Linear Probing, Stochastic Norm, SP-R Sp-Regularization,
DELTA Regularization, BSS Regularization, Co-Tuning
MixUp, MixUp Probability CutMix, Drop-Out, Label Smoothing, Gradient Clipping
Regularization Techniques
Data Augmentation Type (Trivial Augment,
Data Augmentation
Random Augment, Auto-Augmente Auto-Augment Policy*,
Number of operations*. Magnitude*
Optimizer type (SGD, SGD+Momentum,
Optimization
Adam, AdamW, Adamp), Beta-s*, Momentum Learning Rate,
Warm-up Learning Rate, Weight Decay, Batch Size
Scheduler Type (Cosine, Step, Multi-Step, Plateau), Patience",
Learning Rate Scheduling
Decay Rate*, Decay Epochs*
Model
24 Models on the Pareto front (see Appendix 8
In our experiments, we use the tasks contained in the Album benchmark (Ullah et al., 2022) since it contains a diverse set of computer vision datasets. The benchmark is released in three variants with an increasing number of images per dataset: micro, mini, and extended. Concretely, micro has computer vision tasks with fewer classes and fewer images per class than extended. When generating the learning curves, we limited each run to 50 training epochs. As setting a limit is challenging when considering pool of models and tasks with different sizes, decided to constrain the finetuning procedure using global time limit. The configurations trained on the tasks from micro, mini, extended are finetuned for 1, 4, and 16 hours respectively. using a single NVIDIA GeForce RTX 2080 Ti GPU per finetuning task, amounting to a total compute time of 32 GPU months. We summarize the main characteristics of our generated data in Table the Appendix.
6 EXPERIMENTS AND RESULTS
6.1 QUICK-TUNE PROTOCOL
While Quick-Tune finds the best-pretrained models and their hyperparameterss, also has hyperpa- rameters of its own: the architecture, the optimizer for the predictors, and the acquisition function. Before running the experiments, we aimed to design a single setup that easily applics to all the tasks. Given that meta-train the cost and the performance predictor, we split the tasks per Meta-Album version into five folds D = (Di,-. ... Ds) containing an equal number of tasks. When searching for a pipeline on datasets of given fold D., we consider one of the remaining folds for meta-validation and the remaining ones for meta-training. We used the meta-validation for early stopping when meta-training the predictors.
We tune the hyperparameters of Quick-Tune's architecture and the learning rate using the mini ver- sion's meta- validation folds. For the sake of computational efficiency, we apply the same discovered hyperparameters in the experiments involving the other Meta-Album versions. The chosen setup uses an MLP with hidden layers and 32 neurons per layer, for both predictors. We use the Adam optimizer with learning rate of 10-4 for fitting the estimators during the BO steps. We update their parameters for 100 epochs for every iteration from Algorithm| [ Further details on the set-up are specified in Appendix A.2 The inputs to the cost and performance estimators are the dataset metafea- tures (Appendix A.44 and a pipeline encoding that concatenates a categorical embedding of the model m, embedding of the observed curve T(x) and the hyperparameters À (details in Appendix A.5). Finally, for the acquisition function, we use At = 1 epoch as in previous work (Wistuba et al. 2022 since this allows us discard bad configurations quickly during finetuning.

---

Published as a conference paper at ICLR 2024
Performance comparison for Hypothesis Normalized regret, ranks and standard deviations Table 2:
are calculated across all respective Meta-Album (Ulah et al. 2022) subset datasets.
Normalized Regret
Rank
Micro
Mini
Extended
Micro
Mini
Extended
BEIT+Default HP DDfat HP
0.229
0.28 10.100
0.225
2.583
2.61 400
3. 3.
0.223
0.290
0.199
2.500=0.761
2.694g0.264
2.522
DLA+Default HP
0.261_o.000
0.325
0.219
3.062=0.770
3.138 108.0.220
2.977
Qune
0.15340.054
0.139 40.3940.112
0.052 081
1.855441.281
1.555+0.531
1.363 370
Micro
Mini
Extended
100
10%
10
Normalized Regret.
4x20
10-1
2x20
10-1
3000 Wallclock Time (s)
1000
2000
0
5000
10000
15000
0
20000
40000
Wallclock Time (s)
Wallclock Time (s)
Quick-Tune
ASHA
DEHB
BOHB
DyHPO
Random
Figure 3: Comparison against state-of-the-ar HPO methods.
6.2 RESEARCH HYPOTHESES AND ASSOCIATED EXPERIMENTS
HYPOTHESIS QUICK-TUNE BETTER THAN FINETUNING MODELS WITHOUT HYPERPARAMETER OPTIMIZATION.
We argue that ML practitioners need to carefully tune the hyperparameters pretrained models to obtain state-of-the-arr performance. However, due to computational and time limitations, common choice is to use default hyperparameters. To simulate the simplest practical use case, we select three different models from the subset of Pareto-optimal pretrained models (see Fig. 2 i.c. the largest model with the best acccuracy (beit ppei__lrge_pue (urge_patch/66 acc.), the middle model with a competitive accuracy (xcit_ serit_ssall____ 2 p8 __________ 384_88hh (Ali al., 2021 26M and 89.52%), as well as the smallest model with the lowest accuracy among the Pareto front models (dla46x_c (Yu al. 2018); 1.3M and 72.61%) On each dataset in Meta-Album (Ullah et al.) 2022), we finetune these models with their default hyperparameters and compare their performance against Quick-Tune. The default configuration performance, we calculate the average normalized regret (Arango et al. 2021p, computed as detailed in Appendix A. For Meta-Album datasets in a given category, we use the same finetuning time budget, i.e. (micro), 4 (mini), and 16 (extended) hours. As reported in Table 2 Quick outperforms the default setups in terms of both normalized regret and rank across all respective subset datasets, demonstrating that HPO tuning is not only important to obtain high performance, but also achievable in low time budget conditions.
(Bao et al. 2022): 305M parameters, 90.69%
specified in Appendix B.2 To measure the
HYPOTHESIS 2: TUNE OUTPERFORMS STATE-OF-THE-ART HPO OPTIMIZERS.
Gray-box approaches are considered very practical, especially for optimizing expensive architec- tures. We compare Quick-Tuue Tune against four popular gray-box optimizers. ASHA (Li et al. (018).
BOHB (Falkner et al., 2018b), DEHB (Awad et al. 202 and DyHPO (Wistuba et al. 2022). We additionally include Random Search (Bergstra & Bengio 2012b) baseline for a sanity check. The normalized regret is computed for the three Meta-album versions on budgets of 1, 4 and 16 hours. The results of Figure SaaureeBssoo show that proposed method has the best any-time performance compared to the baselines. In an additional experiment, presented in Figure 4 we show that both meta- veeta-taaning and cost-awareness aspects contribute to this goal by ablating each individual component. This
7

---

Published as a conference paper at ICLR 2024
Micro
Mini
Extended
100
10
10"
Normalized Regret
Ex10-
4x10-
3x20m
10-1
2x20
10-1
3000 Wallclock Time (s)
1000
2000
0
5000
10000
15000
40000 Wallclock Time (s)
20000
Wallclock Time (s)
+T++,+ QT:-M.C,+G
QT-M,+CC+
QT:+M,-C++G GP:-M.C.GG
Figure 4: Comparing Qune with (+) and without (-) (Mettalcarriig and (C)ost- -Carees,,,,,s and (G)ray-box optimization. We also compare against DyHPO (=QT-M,-C,+G) and a GP.
Micro
Mini
Extended
10a
10
10a
Normalized Regret
4x10-
10-1
10-1
1000
2000
3000
0
5000
10000
15000
0
20000
40000
Wallclock Time
Wallclock Time
Wallclock Time (s)
24-M
15-M
10-M
5-M
XCIT
DLA
EfficientNet
Figure 5: Varying the model hub size.
behavior is consistent among datasets of different sizes and present in all three meta-dataset versions. We attribute the search efficiency to our careful search space design, which includes both large and small models, well regularization techniques that reduce overfitting in low-data settings such as in the tasks of the micro version. In large datasets, our method finds good configurations faster compared to the baselines, highlighting the importance of aost-awareness in optimizing hyperparameters for large datasets. We additionally compare against a Gaussian Process
(GP) that observes the whole learning curve (At 50), to highlight the necessity of a gray-box approach. In an additional experiment in Appendix C we evaluate our method on the well-known Inaturalist (Horn)
let al. 202T) and Imagenette (Howard) 2019) datasets that are not contained in Meta-A Aabum; there,
our method still consistently outperforms the competitor baselines.
HYPOTHESIS 3: CASH ON DIVERSE MODEL HUBS IS BETTER THAN HPO ON A SINGLE MODEL.
A reasonable question is whether we actually need to consider hub of models at all, or whether perhaps using single, expressive, and well-tuned architecture is sufficient for most datasets. We hypothesize that the optimal model is dataset-specific because the complexities of datasets vary. Therefore, using a single model for all the datasets is sub-optimal practice, and is better to include diverse model hub. Moreover, using model hub allows us to explore cheap models first and gain information about the interactions between the hyperparameters. The information can in turn be leveraged by the predictors when considering larger and more accurate models.
To validate our hypothesis, we select Efficienteer (Tan & Le 2019), X-Cit (Ali et al. 2021 and DLA (Yu al. 2018). These correspond to models with at least 10 evaluated curves in all the datasets and are located on the top, middle, and bottom regions in the Pareto front. Subsequently, we optimize their hyperparameters independently using our Quick-Tune algorithm. We also Quick-Tuma on subsets of 5, 10, and 15 models out of the model hub M with 24 models. The subset of models was created randomly for every dataset before running BO. We execute the optimization the three
8

---

Published as a conference paper at ICLR 2024
Table 3: Comparison against efficient-finetuiing of a single large model.
Hours
24 Hours
Miero
Mini
Extended
Micro
Mini
Extended
Dinov2 LoRA
0.5411
0.049
0.055-2.004
0.33220..08
0.014
0.004
Dinov2 Linear Probing
0.081=0.011
0.081+0.012
0.067=a0.38
0.017
0.012+0.011
QuickTune
0.072
0.039=000014
0.042 016
0.018
0.012
0.003
Micro
Mini
Extended
10a
10%
10"
Normalized Regret
6x10-
3x10-
10
2x10
10-1_
1000
2000
3000
5000
10000
15000
40000 Wallclock Time (s)
20000
Wallclock Time
Wallclock Time
Quick-Tune
LogME + HPO
LEEP + HPO
NCE + HPO
Figure Comparison with two-stage search for models and hyperparameters.
meta-dataset versions for 1, 2 and 4 hours of total budget. Figure demonstrates that, in general, it is better have a pool of diverse models such as 24 models (24-M) or 15 models (15-M), than tuning small set of models or even unique model. Interestingly, we note the larger the dataset is, the larger the model hub we need.
Quick-Tune vs. Efficient Finetuning of a Single Large Model. Although we propose to use model hubs, practitioners also have the alternative of choosing large pretrained model from outside or inside the hub. We argue that a large pretrained model still demands HPO (Oquab et al. 2023 and imposes higher load on computing capabilities. To demonstrate that Quick-Tune still outperforms the aforementioned approach, we compare our method against efficient finetuning approaches of Dinov2 which features BB parameters by: i) finetuning only the last layer of Dino v2, which represents a common practice in the community, and ii) finetuning with LoRA (Hu et al. 2021), parameter- efficient finetuning method. Our results in Table 3 demonstrate that CASH on model hubs via QuickTune attains better results for the different dataset versions.
Quick-Tune vs. Separated Model and Hyperparameter Optimization. We compare Quick- -une with two-stage alternative approach where, we first select a model with its default hyperparameters using state-of-the-an model selection methods, such as LogME (You et al. 2021a). LEEP (Nguyen) et al. 2020) and NCE (Tran et al.] 2019b). Then, we conduct second search for the optimal hyperparameters of the model selected in the first stage. The results reported in F that Quick-Tun outperforms this two-stage approach, thus highlighting the importance of performing combined HPO and model selection.
7 CONCLUSION
We tackle the practical problem of selecting model and its hyperparameters given pool of models. Our method QuickTune leverages gray-box optimization together with meta-learned cost and performance predictors a Bayesian optimization setup. We demonstrate that QuickTune outperforms common strategies for selecting pretrained models, such using single models, large feature extractors, or conventional HPO tuning methods. In addition, we present empirical evidence that method outperforms large-scale and state-of ofie--ofthhe --thh-art art transformer backbones for computer vision. As a consequence, Quick Tune offers a practical and efficient alternative for selecting and tuning pretrained models for image classification.

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGEMENT
Robert Bosch GmbH is acknowledged for financial support. We also acknowledge funding by the Deutsche Forschungsgemeinscaat (DFG, German Research Foundation) under SFB 1597 (Small- Data), grant number 499552394, the support of the BrainLinks- BrainTools Center of Excellence, and the funding of the Carl Zeiss foundation through the ReScaLe project. This research was also partially supported by the Deutsche Forschungsgemeinschatf (DFG, German Research Foundation) under grant number 41 7962822, by the state of Baden- Württemberg through bwHPC, and the German Research Foundation (DFG) through grant INST 39/963-1 FUGG, by TAILOR, project funded by the EU Horizon 2020 research, and innovation program under GA No 952215, and by European Rescarch Council (ERC) Consolidator Grant "Deep Learning 2.0" (grant no. 101045765), Funded by the European Union. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or the ERC. Neither the European Union nor the ERC can be held responsible for them.
Funded by European Union
10

---

Published as a conference paper at ICLR 2024
REFERENCES
S. Abnar, M. Dehghani, B. Neyshabur, and H. Sedghi. Exploring the limits of large scale pre-training. In Proc. of (CLR'22, 2022.
Alaacldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014-20027, 2021.
Sebastian Pineda Arango, Hadi S. Jomaa, Martin Wistuba, and Josif Grabocka. Hpo-b: A large-scale reproducible benchmark for blx hpo based on openml, 2021.
N. Awad, N. Mallik, and Hutter. DEHB: Evolutionary hyberband for scalable, robust and efficient Hyperparameter Optimization. In Proc. of 11 2147-2153, 2021a.
Noor H. Awad, Neeratyoy Mallik, and Frank Hutter. DEHB: evolutionary hyherband for scalable, robust and efficient hyperparameter optimization. CoRR, abs/2105.0988 2021b.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: BERT pre-training of image transformers. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: http:::ttp/ppeeetet httttp:://openrvieeee... net fortt/ffrmmmiio BhZSz59041
Y. Bao, Y. Li, S.-L. Huang, L. Zhang, L. Zheng, A. Zamir, and L. J. Guibas. An information-theorett approach to transferability in task transfer learning. In 2019 IEEE International Conference on Image Processing, ICIP 2019, Taipei, Taiwan, September 22-25, 2019, pp. 2309-2313, IEEE, 2019.
Bergstra and Y. Bengio. Random search for hyper-parameter optimization. 13:281-305, 2012a.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Mach. Learn. Res., 13:281-305. 2012b.
D. Bolya, R. Mittapalli, and J. Hoffman. Scalable diverse model selection for accessible transfer learning. In Proc. of NeurlPS pp. 19301-19312, 2021.
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. arXiv preprint arXiv:1904004233 2019a.
X. Chen, Wang, F., M. Long, and J. Wang. Catastrophie forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning. In Proc. of NeurlPS 19, pp. 1906-1916, 2019b.
Y. Cui, Song, C. Sun, A. Howard, and Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In Proc. of CVPR" 18, pp. 4109-4118, 2018.
S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient Hyperparameter Optimization at scale. In Proc. of ICML''1, pp. 1437-1446, 2018a
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: robust and efficient hyperparameter optimiza- tion at scale. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmmissan, Stockholm, Sweden, July 10-15 2018, pp. 1436-144 1445, 2018b.
M. Feurer, J. Springenberg, and F. Hutter. Initializing Bayesian Hy Hyperparameter Optimization via meta-learning. In Proc. of AAAI'15, pp. 1128-1135, 2015.
Grant Van Horn, Oisin Mac Aodha, and Serge Belongie. inaturalist competition datasets. https
Cgg.thu...co//isspppeaaaaaa/a comp 2021.
Jermey Howard.
2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu
Chen. Lora: Low-rank adaptation large language models. CoRR, abss2200600968, 2021. URL
Pettss///arian wtw.v/orgggaaooo /010/./////...../..////. 2106.09685

---

Published as a conference paper at ICLR 2024
F. Hutter, H. Hoos, and K. Leyton- Brywoo. Sequential model- optimization for general algorithm configuration. In Proc. of LION'II, pp. 507-523, 2011.
F. Hutter, L. Kotthoff, and J. Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019. Available for free at http://atool..org/beatt
H. Jomaa, L. Schmidth-Thieme, and J. Grabocka. Dataser2vec: Learning dataset meta-features. Data Mining and Knowledge Discovery, 35:964-985, 2021.
Donald R. Jones, Matthias Schonlau, and William Welch. Efficient global optimization of expensive black-box functions. J. of Global Optimization, (3(4))455-499) dec 1998. ISSN 0925-5001.
Bo Fu Junguang Jiang, Baixu Chen and Mingsheng Long. //github. aacom/thum
Leannffr--arrinn-iiiiea 2020.
https
Arlind Kadra, Maciej Janowski, Martin Wistuba, and Josif Grabocka. Scaling laws for hyperparameter
optimization. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL
net. worummiiiii gh2uuuGG oha.a
A. S. Khazi, S. Pineda Arango, and J. Grabocka. Deep ranking ensembles for hyperparameter optimization. In The Eleventh International Conference on Learning Representations, 2023, URL
htttps://openrevieet net Forum forrmm2id=_ruvo2K11..
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L., Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P. Dollár, and R. Girshick. Segment anything. arXiv:2304.0268 2023.
A. Kolesnikov, L.. Beyer, X. Zhai, J. Puigcerver, J. Yung. S. Gelly, and N. Houlsby. Big transfer (bit): General visual representation learning. In Proc. of ECCV'20, pp. 491-507, 2020.
Z. Kou, K. You, M. Long, and J. Wang. Stochastic normalization. In Proc. of NeurlPS 2020.
Y. Lee, A.S. S. Chen, Tajwar, A. Kumar, H. Yao, Liang, and C. Finn. Surgical fine-tuning improves adaptation to distribution shifts. CoRR, abs/2210.11466 2022. URL htt.:ttst//....rr////
48550/arXiv. 2210 11466
H. Li, P. Chaudhari, H. Yang, M. Lam, A. Ravichandran, R. Bhotika, and S. Soatto. Rethinking the hyperparameters for fine-tuning. In Proc. of ICLR'222, 20, 2020.
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: handit-based configuration evaluation for Hyperparameter Optimization. In Proc. of IC 17, 2017.
X. Li, Y. Grandvalet, and F. Davoine. Explicit inductive bias for transfer learning with convolutional networks. In Proc. of ICML' 18, pp. 2830-2839 2018.
Xingjian Li, Haoyi Xiong, Hanchao Wang. Yuxuan Rao, Liping Liu, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URLAttts://opeeeeeeee
httt/oormttootooooo 2id= dedddddrrkwss.. rkgbwsAcYm
B. Liu, Cai, Y. Guo, and X. Chen. Transtailor: Pruning the pre- pre-trained model for improved transfer learning. In Proc. of AAAI'21, pp. 8627 8634, 2021.
C. V. Nguyen, Hassner, M. W. Seeger, and Cédric Archambeau. LEEP: transferability of learned representations. In Proc. of ICML 20, volume 1 119, pp. 7294-7305, 2020.
new measure to evaluate
M. Oquab, T. Darcet, Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P.-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
E. Öztürk, F. Ferreira, H. S. Jomaa, L. Semidth- Thietthhhhhmm... J. Grabocka, and F. Hutter. Zero-shot automl with pretrained models. In Proc. ICMI. 22, pp. 1135-1135, 2022.
12

---

Published as a conference paper at ICLR 2024
Sebastian Pineda Arango and Josif Grabocka. Deep pipeline embeddings for automl. In Pro- ceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 23, pp. 1907-1919, New York, NY. USA. 2023. Association for Computing Machinery. ISBN 9798400701030.. doi: 10.1 (0.1145735800353359933333 URL htt.htts:://bi..rr/////// 3580305. 3599300
A. Radford, J. Wook Kim, C. Hallacy, Aditya Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
R. Ramesh and P. Chaudhari. Model zoo: A growing brain that learns continually. In Proc. of ICLR'M 22, 2022.
C. Rasmussen and C. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006.
D. Salinas, H. Shen, and V. Perrone. A quantile-based approach for hyperparameter transfer learning. In Proc. of ICML 220 pp. 8438 8448, 2020.
David Salinas, Matthias Seeger, Aaron Klein, Valerio Perrone, Martin Wistuba, and Cedric Ar- chambeau. Syne tune: A library for large scale hyperparameter tuning and reproducible re- search. In International Conference on Automated Machine Learning, AutoML 2022, 2022. URL https: fttts://prrceedinnge m press/r.esssa www.////aaaaaaaaaaaaaaaaa htm_
K. Schürholt, B. Knyazev, X. Giró-i-Nieto, and D. Borth. Hyper-representations for pre-training and transfer learning. CoRR, abs/2207.10957 2022.
K. Schürholt, D. Taskiran, B. Knyazev, X. Giró-i Nieto, and D. Borth. Model zoos: dataset of diverse populations of neural network models. In Thirty-Sixth Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2022.
G. Shala, A. Biedenkapp, F. Hutter, and Grabocka. Gray-box gaussian processes for automated reinforcement learning. In ICLR 2023, 2023a. URL URL httptsts///peeeee.leeee net/ forum?
Id=rmoMvptXK 7M
G. Shala, T. Elsken, F. Hutter, and J. Grabocka. Transfer NAS with meta-leamed bayesian surrogates. In ICLR 2023, 2023b. URL htt.htttsss/ppeeeeeiieee http::/.eeeeooooeeeeeeeee www.:neee.ooommeeeee www.mmmmraaaaaaaaaaaaaaa
Y. Shu, Z. Kou, 7. Cao, J. Wang, and M. Long. Zoo-tuning: Adaptive transfer from A zoo of models.
In Proc. of ICMI''1, volume 139, pp. 6626-9637, 2021.
Y. Shu, Z. Cao, Z. Zhang, J. Wang, and M. Long. Hub-pathway: Transfer learning from A hub of pre-trained models. 2022.
Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M. Patwary, Prabhat, and R. Adams. Scalable Bayesian optimization using deep neural networks. In Proc. of ICML I55, pp. 2171-20 1181, 2015.
J. Springenberg, A. Klein, S. Falkner, and F. Hutter. Bayesian optimization with robust Bayesian neural networks. In D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett (eds.). Proc. of NeurlPS' 16, 2016.
Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Inter- national Conference on Machine Learning, ICML. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 Proceedings of Machine Learning Research, pp. 6105-6114. PMLR, 2019. URL BRLLttt::/ppccceiinngg mir.
www..................... wwwppppwwss...... wttps////v///tanaaaaam html
C. Thornoon, F. Hutter, H. Hoos, and K. Leyton- Hyperparameter Optimization of classification algorithms. In Proc. of KDD'13, pp. 847-855, 2013.
combined selection and
A. T. Tran, C. V. Nguyen, and T. Hassner. Transferability and hardness of supervised classification tasks. In Proc. of ICCV'19, pp. 1395-1405. IEEE, 2019a.
13

---

Published as a conference paper at ICLR 2024
Anh T Tran, Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classifi- cation tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1305,1405, 2019b.
Ihsan Ullah, Dustin Carrion, Sergio Escalera, Isabelle M Guyon, Mike Huisman, Felix Mohr, Jan N van Rijn, Haozhe Sun, Joaquin Vanschoren, and Phan Anh Vu. Meta-album: Multi-domain meta-dataset for few-shot image classification. In siitty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL attps://meta-allart github. io/]
H. Wang. Yue, X. Ye, Z. He, Li, and Li. Revisit finetuning strategy for few-shot learning to transfer the emdeddings. In The Eleventh International Conference on Learning Representations, 2023.
Ross Wightman. Pytorch image models.
htttp::/ggttub.ccom wwt.:/ccmm.wwg.ge.. rwt.ttmang/hhg////ooomo/mmm
2019.
J. Wilson, F. Hutter, and M. Deisenroth. Maximizing acquisition functions for Bayesian optimization. In Proc. of NewrlPS' 18, pp. 741-749, 2018.
Winkelmolen, N. Ivkin, H. Bozkurt, and Z. Karnin. Practical and sample efficient zero-shot HPO. 2XXi:20077.13822 [st, 2020.
M. Wistuba and Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In Proc. of ICLR 21, 202 la.
M. Wistuba, N. Schilling, and L. Schmidt- Thieme. Sequential Model-free Hyperparameter Tuning. In Proc. of ICDM '15, pp. 1033-1038, 2015.
M. Wistuba, N. Schilling, and Schmidt-Thiene Two-stage transfer surrogate model for automatic Hyperparameter Optimization. In Proc. of ECML/PKDD 16, pp. 199-214, 2016.
M. Wistuba, A. Kadra, and J. Grabocka. Supervising the multi- filli-fiielity race hyperparameter configurations. In Proc. of NeurlPS 22, 2022.
Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates.
In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021
M. Wortsman, G. Ilharco, S.r Yitzhak Gadre, R. Roelofs, R. Gontijo Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt. Model soups: averag- ing weights multiple fine-tuned models improves accuracy without increasing inference time. In Proc. of ICML 22, volume 162, pp. 13995-23992 2022a.
M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. G. Lopes, Hajishirzi, A. Farhadi, H. Namkoong, and L. Schmidt. Robust fine-tuning of zero-shot models. In Proc. of CVPR 22, pp. 7949-7961, 2022b.
J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable features in deep neural networks? In Proc. of NeurlPS' 14, pp. 3320-3328, 2014.
K. You, Z. Kou, M. Long, and J. Wang. Co-tuning for transfer learning. In Proc. of NeurlPS'20, pp. 17236-17246, 2020.
K. You, Y. Liu, J. Wang. M. Jordan, and M. Long. Ranking and tuning pre-trained models:
A new paradigm of exploiting model hubs. CoRR, abs/2110.10545, 2021a. URL https
//arxiv. wwtp/rggggaaa.ooooeeeeo
K. You, Y. Liu, J. Wang, and M. Long. Logme: Practical assessment of pre-trained models for transfer learning. In Proc. ICMI.'11, pp. 12133-12143, 2021b.
Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 2403-2412, 2018.
Zhong, X. Wang, Z. Kou, J. Wang, and M. Long. Bi-tuning pre-trained representations. arXiv preprint arXiv:201 1.06182, 2020.
14

---

Published as a conference paper at ICLR 2024
A ALGORITHMIC DETAILS
A.1 NORMALIZED REGRET
Given observed performance y, the normalized regret is computed per datasets follows:
Ymax
Unorm
Umax
Mmiin
where Vmnax: Umin in Equation 7 are respectively the maximum and minimum performances in the meta-dataset.
A.2 ADDITIONAL. SPT-UP DETAILS
The categorical encoder of the model is a linear layer with output neurons, while the learning curve embedding is generated from convolutional neural network with two layers. For the of the hyperparameters of the deep-kernel Gaussian process surrogate and the acquisition function, we followed the settings described in the respective publication (Wistuba et al., (022) and always used the setup suggested from the authors unless specified otherwise. We use the Synetune library (Salinas al. 2022) for the implementation of the baselines.
A.3 META-TRAINING ALGORITHMS
We present the procedure for meta-traine meta-training the cost and loss predictors in Algorithm 2 Initially, we sample random batches after choosing a dataset randomly within our meta-dataset, and then we update the parameters of each predictor that it minimizes their respective losses. The same strategy used when updating during BO but with fewer iterations. We meta-train for 10000 iterations using the Adam Optimizer with a learning rate of 0.0001.
Algorithm 2: Meta- Meta-training training Algorithm
Input: Metadata with precomputed losses and cost H(M)) with datasets D = -dd,,..,NNk
learning rate yı, Epochs F
Output: Meta-learned parameters 0,
Random initialize parameters 8, for loss predictor Ê(-) and cost predictor é(-);
for i {1...E) do
Sample dataset index i ~ U11, [D]] and its metafeatures d;:
Define the subset of history associated to d,: H4 H(M),
t, ((x,t,l(x,t,, ((,,,,,, Vo ((....(....))))))). logg (((x,t,d) di) x, d(
Compute 50
Compute 87 V ((,,,dddddd) ((xx,td;) è (x, (x,
Update parameters 0 = 0 - μ 50, - 1..67
end
g(af) f- 0, (A1) +- 7
10 return ((M) (15)
A.4 META-FEATUREE
Similar to previous work (özürkk et al. 2022), we use descriptive meta-features of the dataset: number of samples, image resolution, number of channels, and number classes. Any other technique for embedding datasets is compatible and orthogonal with approach.
A.5 PIPELINE ENCODING
Our pipeline encoding is the concatenation of the hyperparameters A, the embedding of the model name Emodel(mm)) and the embedding of the learning curves. Given the performance curve 7 =(x;, ().
15

---

Published as a conference paper at ICLR 2024
obtain the respective embedding (T using 2-layer convolutional networks following a similar setup from previous work (Wistuba et al. 2022). For obtaining the model embedding, we and feed this representation into linear layer (Pineda Arango & Grabocka 2023). The pipeline encoding is finally defined as:
transform the model names into one-hot-eneonn
Pipeline (xcodig(x;) = [Ms moddel Epert (T (x)
(8)
The parameters of the encoders Emodal (-art()) are jointly updated during meta-training and while fitting the predictors during BO.
B META-DATASET DETAILS
B. META-DATASET COMPOSITION DETAILS
While generating the meta-dataset, we take into account the dependencies of the conditional hy- perparameters. Every triple (model, dataset, hyperparameter configuration) resulted in a finetuning optimization run that produced a validation error and cost curves. A few of the combinations infeasible to evaluate due to the model size, thus some triples can have fewer evaluations. For instance, some pipelines failed due to the GPU requirements demanded by the number of parameters of the model and the number of classes of the datasets. In that case, we decreased the batch size iteratively, halving the value, until it fits the GPU. In some cases, this strategy was enough, thus some models have more evaluations than others. In Table we present the list of datasets per and indicate the heavy datasets with (+), ie. with a lot of classes or lot of samples in the extended version. The majority of the datasets are present in all three versions of Meta-Album, except the underlined ones, which are not present in the extended version. The OpenMI. Ids associated to the datasets are listed in Table Table provides descriptive statistics regarding the generated meta-dataset for every corresponding Meta- Alba-Album version.
Table 4: Datasets per Set in Meta-Album
Set
Dataset Names
BCT, BRD*, CRS, FLW, MD_MMDMMLL PLK** PLT VIL* RESISC, SPT, TEX ACT 40, APL, DOG, INS 2a, MD 5 BIS, MED LE, PLT NET*, PNU, RSICB, TEX DTD ACT_410, AWA*, BTS*, FNG, INS MD 6,
1
2
PRT, RSD*, TEX ALOT*
Table 5: OpenML IDs for Datasets per Split and Version
Version
Set 0
Set
Set
44241, 44238, 44239, 44242,
44313, 44248, 44249, 44314,
44275, 44276, 44272, 44273,
Micro
44237, 44246, 44245, 44244,
44312, 44315, 44251,, 44250,
44278, 44277, 44279, 44274,
44240, 44243
44247, 44252
44271, 44280
44285, 44282, 44283, 44286.
44298, 44292, 44293, 44299,
44305, 44306, 44302, 44303,
Mini
44281, 44290, 44289, 44288,
44297, 44300, 44295, 44294,
44308,
44307, 44309, 44304,
44284, 44287
44291, 44296
44301, 44310
44320, 44317, 44318, 44321,
44331, 44326, 44327, 44332,
44338, 44340, 44335, 44336,
Extended
44316, 44324, 44323, 44322,
44330. 44333, 44329, 44328,
44342, 44341, 44343, 44337,
44319
44325
44334
Table 6: Quick-Tune Composition
Meta-Dataset
Number Tasks
Number of Curves
Total Epochs
Total Run Time
Micro
30
8.712
371.538
2.076 GPU Hours
Mini
30
6.731
266.384
6.049 GPU Hours
Extended
26
4.665
105.722
15.866 GPU Hours
16

---

Published as a conference paper at ICLR 2024
B.2 HYPERPARAMETER SEARCH SPACE
Table Detailed Search Space for Curve Generation. Bold font indicates the default configuration.
Hyperparameter
Name
Options
Conditional
Group
Percentage to freeze
0, 0.2, 0.4, 0.6, 0.8, 1 None, 0.65, 0.75 True, False
Layer Decay
Linear Probing
Fine- TuninT
Stochastic Norm
True, False
Strategies
SP-Regularization
0.0001, 0.001, 0.01, 0.1
DELTA Regularization
0, 0.0001, 0,001, 0.01, 0.1
BSS Regularization
0.0001, 0.001, 0.01, 0.1 0, 0.5, 2,
Co-Tuning
MixUp
0.2, 0.4, 2,
MixUp Probability
0.25, 0.5, 0.75,
Regularization Techniques
CutMix
0, 0.1, 0.25, 0.5, 1,2,4
DropOut
0, 0.1, 0.2, 0.3, 0.4
Label Smoothing
0.05, 0.1
Gradient Clipping
None, 1, 10
None, trivial_ aaggmet,, random_ random_augment, auto auooment None, v0, original
Data Augmentation
Data
Augmentation
Auto Augment
Data Augmentation
Number of Operations
2,3
(Random Augment)
Data Augmentation
Magnitude
9, 17
(Random Augment)
SGD, SGD+Momentum, Adam, Ad Adamp
Optimizer Type
Scheduler Type (Adam, Adamw, Adamp)
Optimizer Related
(0.9, 0.999), (0, 0.99), (0.9, 0.99), (0, 0.999)
Betas
0.1.0.01, 0.005, 0.001, 0.0005,
Learning Rate
0,0001, 0.00005, 0.00001
Warm-Up Learning Rate
0.000001, 0.00001
0, 0.0000, 0.0001,
Weight Decay
0.001, 0.01.0.1
Batch Size
2448.16.32.641118255651111
Optimizer Type (SGD+Momentum)
Momeutm
0.8, 0.9, 0.95, 0.99
None, Cosine, Step. Multistep.
Scheduler Type
Plateau
Scheduler Related
Scheduler Type (Plateau) Scheduler Type (Step, Multistep)
Patience
2,5
Decay Rate
0.1, 0.5
Scheduler Type (Step, Multistep)
Decay Epochs
10, 20
Model
Model
See Table
Table / shows the complete search space of hyperparameters. During the curve generation, we sample uniformly among these discrete values. Some hyperparameters conditional, i.e. their only
17

---

Published as a conference paper at ICLR 2024
present when another hyperparameter gets specifie set of values. Thus, also list explicitly which are the conditions for such hyperparameters.
We report the hyperparameters for the default configuration in Experiment by using bold font in Table 7
B.3 MODEL HUB
We list all the models on the Pareto Front from Timm's library as provided on version 0.7.0devO. Moreover, we report their size (number of parameters) and the top-1 accuracy in ImageNet.
Table 8: Models the pareto front
Model Name
No. of Param.
Top-1 Acc.
beit beit_large patch16 512
305.67
90.691
volo_ _____12
296.09
90.610
volo_d5 d5 d5_448
295.91
90.584
volo d4 448
193.41
90.507
swinv2_base window window|2to24__1 22
192t0384222kttlk
87.92
90,401
ceit_base_patchh66
86.74
90.371
volo d3_448
86.63
90.168
tf efficientnet b7
66.35
90.093
convnext_sman ___4__444___ _n__f22
50.22
89.803
tf tf_efficientnet_ b6
43.04
89.784
volo_dl d1 _384
26.78
89.698
@itttsmall_____0 _e___________________ 284_dist
26.21
89.515
Laa___malllllaahhhl_a111 6_166666644__611 184_in21fttl tf efficientnet b4
22.21
89.367
19.34
89.303
xcit ceii__tiy______e __1___4_____ ________________________ 384 dist
12.11
88.778
scit_tiny_12_pp1 1118__34_diii_ dist
6.71
88.
edgenext_snne smalll
5.59
87.504
xcit nant_ 12 122__8_33__ 22_43844__0 384_dist dist
3.05
85,025
mobilevitv2_077 075
2.87
82.806
edgenext_x, x x_sllll mobilevit
2.34
81,897
2.32
81.574
edgenext xx_ xx_small
1.33
78.698
mobilevit_xo xxs dla46x_c
1.27
76.602
1.07
73.632
C ADDITIONAL EXPERIMENT: QUICK-TUNE ON DATASETS OUTSIDE META-ALBUS
Meta-album contains broad set of datasets, ranging from small-size datasets with 20 samples per class and 20 classes, to more than 700 classes with to 1000 samples per class. Moreover, it offers diverse set of domains that foster a strong benchmarking of image classification methods. To further verify the generalization outside the curated datasets present in Meta-Album, we run experiments Albu-.. Initially, we run Quick-Tune on
on two well-known datasets that are not present Imagenette (Howard 2019) by using time budget of 4 hours. Additionally, we run Quick-Tune it on Inaturalist (Horn et al. (021) with a time budget of 16 hours. Finally, we transfer the estimators meta-learned the mini and extended splits respectively. We compare the results to the same gray-box HPO baselines as Hypothesis The selection of the budget and the transferred estimators based the similarity of each dataset size the corresponding Reta-Album super-set.
18

---

Published as a conference paper at ICLR 2024
imagenette2-320
inaturalist
10P
10P
Normal lizeed Regret
10
10-2
10-t
3.
5000
10000
15000
40000 Wallclock Time (s)
20000
Wallclock Time (s)
QuickTune
DyHPO
ASHA
DEHB
BOHB
Figure Evaluation of Quick-Tune on Datasets outside Meta-Album,
D DETAILS ON DINOV2 EXPERIMENT
D.I FINETUNING LAST LAYER DINOV2
common practice is use large feature extractors as the backbone and just train a linear output layer. We argue that selecting the model from pool and optimizing its hyperparameters jointly is more effective approach. Firstly, large backbones are often all all-ppuroose models that may be inferior to model hubs when downstream tasks deviate largely from the backbone pretraining and may require non-trivial finetuning hyperparameter adaptations. such, individual large models may violate the diversity property observed in our third hypothesis above. Secondly, due their large number of parameters, they are expensive to optimize.
Micro
Mini
Extended
10
Normalized Regret:
10 1
10-1
10-2
10
10:2
10-3
10-3
10a
10a
10a
10
10a
10
Wallelock Time
Wallclock
(8)
Wallelock Time (8)
QuickTune
DINOv2
Figure 8: Results for finetuning the last layer of DINOv2. We relax the efficiency conditions by allowing a bigger time budget but still limited to GPU.
To verify that selecting model from model hub with Quick-Tune is more effective approach than using an all-purpose model, we compare to DINOv2 (Oquab et al., 2023) According to the method's linear evaluation protocol, the procedure to classify downstream tasks with pretrained DINOv2 involves performing a small grid search over subset of its finetuning hyperparameters (104 configurations in total including learning rate, number of feature extraction layers, etc.). We adopt this grid search in our comparison, evaluating all the hyperparameter configurations on the grid. For each meta-album version, we then compare the normalized regret against the wall-clock time between DINOv2 and Quick-Tune. For the experiment, we increase Quick-Tune's budget to match DINOY2's ' budget requirements since the evaluation of the full grid of DINOv2 requires more time than previous experiments. In Figure 8 present the results of our comparison, where, method manages to outperform DINOv2 in all the considered benchmarks, highlighting the benefits of our designed search space.
19

---

Published as a conference paper at ICLR 2024
While performing our comparison, small minority of the DINOv2 runs failed due GPU memory limitations, and for a few runs, we had to minimally adjust the DINOv2 default hyperparameter configurations "n last blocks" to adapt to our GPU memory limitation. Tables 9 TO II depict for which of the datasets we ran with the default hyperparameter configurations according to (Oquab) et al. 2023) and which we adapted due the single-GPU constraint (RTX2080). Runs indicated with failed due to GPU memory limitations and for runs indicated by "n_last l___s__blookk = 1" we ran with the default hyperparameters except for the "n la_sas__bia_ _ls__kkkkksks argument that had to be changed from 4 to 1 to fit on the GPU.
Table 9: Subset Micro
Table 10: Subset Mini
Table 11: Subset Extended
Datavet
Linear Fval. Hyp-
Dataset
Linear Eval. Hyp.
micro se00 DCT
micra setill FL.
Micoo_ses0_Cg
settl BRD
DINOv2 defasl
DINOv2 defusit
DINOv2 defesi
DINOV2 defasl
mini set, BCT
mini mini_sei0_CRS
sett) FLW
HKD
a_last blockse
DINOVZ defamil
DINOv2 defoalt
blockss
extended set0 CRS extanded send FLW exiended set0 RESISC estended seit extended sctt TEX
extended sett RCT
Datuset
Linear Eval. Hyp.
n. blockes
n. last. blocke1
bloks-
mieroseeto, seto, set__DD_aa MD_MMDD_LXX MIX
PLK
DENOV2 defusi
DINOx2 defasi
sein MDD MIX
mini_seti PLK
[ast blocks+1
DINOY2 default
nu_bst blocks=
blucks=
micro PLT VIL.
mecrossetl RESISC
micro seth SPT
DINOV2 defosit
DINO<2 defasit
DINOV2 defoslt
sini
muni settlettlltttt RESISC
mini set0 _et__ssPP_SPTT""""T
DINOY2 defanlt
DINOV2 defanit
DINOV2 defoult
esuended set set___CC ACT 40 exiended setl API.
DINO-2 default
last blocke
blocker
micro
DINOV2 defosit
mini, sett TEX
DINOV2 defailt
exeendd DOG
blocks1
micro setl ACT
micro setl LAPL
DINOVZ defaslt
DINO<2 defaslk
mini sett L,T, 40
mini, mini_setl sesi__ee_____.. LAPL
DINOv2 defamlt
DINOYZ default
extended set2 ACT estended wel PTT DOC
DIND defull
DINO-2 defull
micro seci DDG
DINOV2 defusit
DINOV2 defusi
mini sen1 DDGG
I_ bicki-l
a_last blocket
extended_set0 BRD
exuended PL.K
micro setl MD BIS
microsett MED LP
DINO defull
DINOV2 defuslt
mini set1
mini set MED LF
RIS
n. bloki-
DINOV2 defonlt
essended_seti J[S_ exlended set MED 1F
extended sett PLT VII.
mitro_setl_P Met NET
micro PNU
DENOV2 defusit
DINO defuil
mini sott PPP_ NET
PNU
DINOY2 defanlt
DINOv2 defmil
extended PT, NET
n lus blocks
micro_setl _ee__set___E_______ DID
micro_serl setl RSIC
DINOV2 defuslt
DINO<2 defaslt
mini_setl Sese__eT______eee DED
mant sett RSICB
DINOV2 Gefanit
DINOV2 defanlt
exsended seil RSICB
extended PNU
nu blocks=
micro set2 ACT 410
micro, se12 FNG
micro RTS
DINO defusl
DINO defoult
DINOV2 defoslt
DINOVE defusit
mini se12 ACT 410
mini sa12, FNG
mini s012
DINOv2 defailt
DINOV2 defamlt
DINOV default
DINOV2 defanit
cxiendedsstti_ _TE___EX____ DID
estended Baciedd_st2___000_ AWA
extended FNG
extended set2 BTS
Iust blockas] nulst blocks=
blockss
micro ser2 INS
DENOVZ defonlt
mini __t2__222_1 INS
Ia biocks1
extended PRT
se12 MD
DINO defall
mäni mini PLT
blckel
estendel
micro sr22 IXC
micm sa2 RSD
micro se______PP PKT
DINOV2
DINO defull
DINOV2 defoslt
mini set?. RSD)
mini_sefl, PKI
DINO defailt
DINOV2 defanit
DINOV2 defuilt
extended set2 Betee___e____XXAA____ ALOTT
essended see2_eNS
nu blocks
TEX ALOTT
DINOV2 defonit
mini, ser2 TEX
blockel
D.2 DINOV2 EFFICIENT FINETUNING WITH LORA
As an alternative, instead of the infeasible full finetuning, we finetune DINOv2 with LoRA (2), which is state-of-the-art method for finetuning transformer models (such DINOv2). LoRA demands ca. 1% of the parameters of the full finetuning strategy and fits into our GPU restrictions for most of the datasets. Furthermore, LoRA reported to achieve similar or better performance compared to full finetuning [2].
We stress that even with LoRA, the large DINOv2 model (1B params) exceeds our GPU memory capacity in some datasets. Thus, we present the results of the experiments in Table with the datasets that DINOv2 were successfully trained, namely: 30 datasets for micro, 20 datasets for mini, and datasets for extended. We report results for 4 Hours (4H) and 24 Hours (24H) of total budget where QuickTune outperforms both alternatives of finetuning DINOv2.
20


Reviews:
[{"review_id": "O2wq7FFmH7", "reviewer": "Reviewer_qaoF", "summary": "This paper introduces a Bayesian meta-learning approach called Quick-Tune to jointly optimize choice of models and hyperparameters for finetuning, i.e. a Combined Algorithm Selection and Hyperparameter Optimization (CASH) approach.  Quick-Tune builds on top a previous grey-box hyperparameter optimization approach called DyHPO with a cost-aware acquisition function and a meta-learning approach to warmstart loss and cost estimators.  Experiments on the Meta-Album benchmark for few-shot learning optimizing over a search space including models from TIMM shows Quick-Tune to efficiently select models and associated hyperparameters for new tasks, exceeding other standard hyperparameter methods as well as two-stage model selection and hyperparameter tuning baselines.  As part of training QuickTune, the authors also collect a meta-dataset of learning curves with associated hyperparameters and datasets to add to the set of meta-learning benchmarks.", "questions": "- What is the unnormalized performance of the approaches in Figure 3 and Figure 4?\n- Instead of rank, how would Figure 1 look as a heatmap of unnormalized performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Paper is well written and clear.\n- Quick-Tune addresses an important problem of how to efficiently select and tune models from a model hub for finetuning/transfer learning on a new dataset.\n- Experiments appear to be thorough and high quality.", "weaknesses": "- In the ablation study, for the Micro and Mini cases, QT: -M, +C, +G (DyHPO with cost-awareness) performs as well as QT: +M, +C, +G (DyHPO with cost-awareness and meta-learning) which shows most of the benefit coming from the cost-aware aspect of QuickTune and not the meta-learning.\n- Novelty is limited since the core of Quick-Tune is DyHPO, a prior HPO method.  Cost-aware acquisition functions have been used in the past and the approach of using meta-features and meta-learning good initializations for the estimators also lack originality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mUYpJZFPz4", "reviewer": "Reviewer_LquR", "summary": "* A challenge when in the  pretraining (PT) / finetuning (FT) paradigm is deciding (1) what PT model to use for a given task and (2) what hyperparameters to use when FT it. This paper presents a method to identify the best pretrained model and finetuning hyperparams to use for a new dataset, using bayesian optimization/\n* The proposed method first pretrains surrogate models on a large meta-dataset of finetuning pipeline runs, which captures variation in datasets, model architectures.\n* These surrogates are then used to define an acquisition function that defines how to select a finetuning pipeline (model specification and hyperparameter set) during each step of Bayesian optimization. Once more data is acquired, the surrogates are also updated.\n* The specific acquisition function is a variation of expected improvement, including a term that captures the cost of running the finetuning pipeline (as opposed to being based purely on performance alone).\n* In experiments, the surrogates are trained on a large set of learning curves from the timm benchmark. The algorithm is then applied to the Meta-Album dataset, and the results demonstrate that the proposed method outperforms baselines.\n* Ablations where the meta-training step and cost-aware acquisition are removed demonstrate that both parts are important.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* Interesting problem choice, and original direction (to the best of my knowledge)\n* Presentation, technical detail, and experiments are good quality.\n* Proposed method has strong performance  on benchmarks considered\n* Ablations of the cost-aware component of the acquisition function and meta-training were informative.", "weaknesses": "Paper is strong and has thorough results. The one thing I was curious about was how the method performs on other standard benchmarks such as Imagenet, and whether any of these results can be validation on different domains (eg text datasets, where finetuning is also very common).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5oFQSr9NHx", "reviewer": "Reviewer_xZL3", "summary": "This work presents and evaluates an up-to-date combined model and hyperparameter selection method for transfer-learning/finetune setup. It works based on observing the learning curves of training runs and iteratively select which run to continue further based on bayesian optimization. The model maintains a performance predictor and a cost predictor, which is being updated as the search proceeds. The model parameters are/can be meta-learned.\n\nThe method is evaluated on a search space composed of 24 models on a pareto curve of number of parameters and ImageNet accuracy and on 86 tasks. The hyper parameter search space includes relevant settingslike finetune strategies, regularisation, data augmentation, optimizers, learning rate schedule.\n\nThe method is compared against other approaches such as: single model + HPO and model selection + HPO for selected model.", "questions": "What splits are being used to train, guide search and report test performance? It would be good to have that explicit in the text.\n\nHow is normalised regret calculated?\n\nHow do the curves in figure 3 look like when the wallclock time gets extended?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Model selection and hyperparameter optimization are practical problems many ML practitioners encounter. It is welcome to see a method able to tackle both together and that doing so provides benefits from doing it step wise. This reviewer is not aware of model selection papers / transferability estimation doing joint optimization in a recent finetune/transfer setting as in here.\n\nThe paper setting with 24 models and 86 datasets from Meta-Album when training for up to 1,4,16 hours seems reasonable and one practitioners can relate to.", "weaknesses": "I do not follow how the normalised regret is calculated. In particular how is y_max and y_min calculated? Is it provided by Meta-Album datasets? Is it the min/max of all runs ever done on this study? How significant is a 10% regret and is there any more expensive way to close that gap when using this approach?\n\nThe curves on plots like figure 3, shows a different behaviour between the methods. It is hard to predict if quick-tune always beats them or if that story changes as the wallclock time gets extended. It would be interesting to see if the search approaches regret 0 or stays ~10% above it.\n\nOne issue with model selection in particular with small datasets is overfitting, including to the validation set. I expected some discussion around this and also an explicit reference to which splits are used during which phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KJtC4VoP5D", "reviewer": "Reviewer_omMr", "summary": "Currently there are many models available online but it is challenging to decide which of them to select and how to fine-tune it to the downstream task. The paper focuses on this challenge and introduces a method to find a suitable pretrained model together with suitable hyperparameters for fine-tuning. The authors construct a meta-dataset and use it for meta-learning a performance predictor that can jointly select the model and the hyperparameters.", "questions": "* There are micro, mini and extended versions of the Meta-Album datasets, which poses the question if Quick-Tune could benefit from using the smaller subsets for more quickly finding settings that work well on the larger subsets. It would be interesting to study this to see if it works for Quick-Tune and how large speedups can be obtained.\n* How well does the method work on other downstream tasks or data modalities?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper studies an interesting problem setting that is practically important and likely to further increase in importance. The trend is that there will be more and more pretrained models available and these will be more commonly used, so a method for deciding how to select a suitable one and fine-tune it is valuable.\n* The method seems to be novel and appears to be a well-designed solution to the presented problem. It also offers strong performance compared to the baselines that are evaluated.\n* The paper is well-written and easy to read. The figures and tables are well-presented and make it simpler to understand the work and the results. The questions that are studied are clearly stated.\n* The paper contributes a meta-dataset that can be interesting for other researchers to study this important problem.\n* A variety of relevant research questions are asked and answered reasonably well.", "weaknesses": "* The paper only considers computer vision classification, while related works such as LogMe (You et al., ICML’21) consider various downstream tasks (classification and regression), and modalities (vision and language). It can be impractically expensive though to evaluate the approach on also these other settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)