MASIF: Meta-learned Algorithm Selection using Implicit Fidelity Information
Authors: Tim Ruhkopf
Aditya Mohan
Difan Deng
Alexander Tornede
Frank Hutter
Marius Lindauer
Venue: Accepted by TMLR ()

Abstract: Selecting a well-performing algorithm for a given task or dataset can be time-consuming and
tedious, but is crucial for the successful day-to-day business of developing new AI & ML
applications. Algorithm Selection (AS) mitigates this through a meta-model leveraging
meta-information about previous tasks. However, most of the available AS methods are
error-prone because they characterize a task by either cheap-to-compute properties of the
dataset or evaluations of cheap proxy algorithms, called landmarks. In this work, we extend
the classical AS data setup to include multi-fidelity information and empirically demonstrate
how meta-learning on algorithms’ learning behaviour allows us to exploit cheap test-time
evidence effectively and combat myopia significantly. We further postulate a budget-regret
trade-off w.r.t. the selection process. Our new selector MASIF is able to jointly interpret
online evidence on a task in form of varying-length learning curves without any parametric
assumption by leveraging a transformer-based encoder. This opens up new possibilities for
guided rapid prototyping in data science on cheaply observed partial learning curves.

Published in Transactions on Machine Learning Research (3/2023)
MASIF: Meta-learned Algorithm Selection using Implicit Fidelity Information
Tim Ruhkopf
trrukkopf@ai. www.://oo..uiiiooooor...dd
Institute of Artificial Intelligence Leibniz Universily Hannover
Aditya Mohan
a. a.mohan@ai.com www.:///wwaa..u-mmmnnneee..eoo
Instibute of Artificial Intellgence Leibniz University Hannover
Difan Deng
d.deng@ai. wwn.uuaoi.uni-eeemveee...dddd
Institute of Artificial Intelligence Lesbniz University Hannover
Alexander Tornede
Institute of Artificial Intelligence Leibniz University Hannover
Frank Hutter
floßess ciliesssni-freibrr.. de
Machine Learning
Albert- Albert-Luduigs University Freiburg
Bosch Center for Artificial Intelligence
Marius Lindauer
m.lindauer@ai. uni-hunnovvee.deeeee
Institute of Artificial Intelligence Leihniz University Hannover
Reviewed on OpenReview: https: // openeevee......... net/ forum? id 5aYGXaByI6
Abstract
Selecting well-performing algorithu for a given task or dataset can be time-consuming and tedious, but is crucial for the successful day-to-day business of developing new Al & ML applications. Algorithm Selection (AS) mitigates this through meta-model leveraging mela-informatioon about previous tasks. However, most of the available AS methods are error-prone because they characterize a task by either cheap-to-compp dataset or evaluations of cheap proxy algorithms, called landmarks. In this work, we extend the classical AS data setup to include multi-fidelity information and empirically demonstrate how meta-learning on algorithms' learning behaviour allows us to exploit cheap test-time evidence effectively and combat myopia significantly. We further postulate budget-regret trade-off w.r.t. the selection process. Our new selector MASIF is able to jointly interpret online evidence task in form of serying-lengtle learning curves without any parametric assumption by leveraging a transformer-basee encoder. This opens up new possibilities for guided rapid prototyping in data science on cheaply observed partial learning curves.
properties of the
1 Introduction
Data scientists typically spend most of their time with data engincering and thus often have less time for choosing well-performing algorithm for given task (Anaconda| 2020). This selection process is typically powered by both the experience of the data scientist w.r.t. the performance of algorithms on previous tasks

---

Published in Transactions on Machine Learning Research (3/2023)
well limited number of partial evaluations of the candidate algorithms on the task at hand. In doing so, data scientist inherently carefully trades off (i) time invested into actively gathering performance information on the new task to make an informed decision and (ii) the regret in terms of performance incurred by possibly not selecting the best algorithm.
On one extreme end of this trade-off lie classical AS (Rice: 1976 Kerschke et al. 2019) approaches which characterize a task using handerafted meta-features (Vanschoren) 2018) of the corresponding dataset. These features are supposedly informative w.r.t. the performance of an algorithm used to learn which candidate to recommend for new task. In practice, such dataset meta-feetures may show limited association with the performance of the considered classes of algorithms (Pfahringer et al. 2000; J. Fürnkranz 2001). In particular, so-called dataset meta-features, corresponding to cheaply-compulall properties of the dataset (including simple, statistical, information-theeorttes complexity- and model- based properties (Vanschoren 2018)): are known to be only indicative of algorithm performance to a limited degree (Pfahringer et al.) 2000). Similarly, the correlation of landmarking mota-features, ic., performances of cheap proxy algorithms computed on the new dataset, can be limited, as they hinge on how well the proxy algorithms' performances are associated with those of the pool of candidates. For instance, the performance of a decision stump might not be informative on how well a deep neural network will perform.
that particular dataset and
On the other extreme end of this trade-off lie existing learning curve approaches (Mohr & van Rijn 2022
which exclusively invest time into gathering premature performance approximations on the candidates and seek to extrapolate off of it. Examples of these approximations are validation performances of models trained for a limited amount of epochs or only on parts of the training data. As such, learning curve approaches invest comparatively more budget on obtaining information naturally associated with the final performance of the classes of algorithms. A common downside to these approaches lies in their extrapolation strategy. Observing only premature approximations of the performance data with little or no meta-knowlede about an algorithm's 's learning behavior, learning curve methods often resort to strong parametric assumptions (Mohr & van Rijn 2022) and are myopic in the sense that an extrapolated estimate of the final performance can only be based on the observed premature performance values. Further, with few exceptions, such as (van Rijn let al. 2015 Klein et al., 2017b Baker et al. 2018 Long et al.) 2020). they only extrapolate each algorithm independently, ignorant of possible existing relations in learning behaviors to other algorithms.
In this work, we propose MASIF (Mela-learned Algorithm Selection using Implicit Fidelity H an approach designed to support data scientists in the AS process in a much more intuitive manner. It supports a data scientist's 's active discovery process on a new dataset, as it allows refining the expectation on the final ranking over algorithms based on their incremental budget allocation. This provides the user with full flexibility in different trade-offs between invested budgets and regret of the final ranking. Tt is however also a limitation in the sense that a user has make this decision. As alleviation, we later provide an exemplary solution this but note that it opens up new area for research.
MASIF is powered by a transformer-baseed model accounting for the sequential nature of learning curves and interpreting their (partial) informational content. It learns to interpret them based on the meta-knowledge w.r.t. the candidates' learning behavior oblained from observing them on other datasets. Using this encoder and the available keoweeee our model neither requires explicitly extrapolating the curves nor does it necessitate assumptions on their paramctric shapes. Learning latent representation of cach curve respectively, a subsequent transformer and MLP merge the gathered test-time evidence in the form of observed premature performances. This allows learning cross-correlations between the algorithms' learning behaviors (cf. Figure[]
Overall, we make the following contributions:
1. From practical perspective, we formalize a meta-learning multi-fidelity setup that closely resembles a data scientist's workflow. Based on this we discuss the budget-regret trade-off that contemporary algorithm selectors implicitly make.
2. We introduce MASIF, a novel learning curve interpreter living in this framework, that leverages both dataset meta-features and formulates a well-informed meta-prior on the candidates' learning
MASTF's code published on http://aaarnuraee.pppeecccceee/s
2

---

Published in Transactions on Machine Learning Research (3/2023)
Qmets-trainn
Qmats-teat
Algorithon A
LC pptrain Ar.
A
MASIP
R R
A horithm
ALAi
LCD garetn Aip
Algorithon Selection
Meta- Feature
9D peratty
Drest
RAs
Dataset
Dataset Dpptrinn
Dataset Drxst
Figure In contrast classical algorithm selection trained only on meta-fcatures, MASIF utilizes the information from both meta-features and learning curve values to predict the best performing algorithms.
behaviors. It mitigates the fallacies of both, classical AS and learning curve-based approaches by interpreting multiple learning curves of varying lengths and, utilizing meta-knowledge to combat. myopia leading better-informed ranking with less compute a new dalaset.
3. We propose new evaluation scheme based on fidelity slices suited for demonstrating the aforemen- tioned tradc-off.
4. We demonstrate the usage of novel learning curve interpreter algorithm selector based user's budgetary and regret preferences, and evaluate its performance against other baselines several benchmarks.
2
Preliminaries
As we seek to unify classical AS, multi-fidelity, and learning curves, we re-examine them and offer a notation, with which we later detail how selectors behave. Particularly, this notation highlights both the available kootledeo and the associated cost of acquiring information on a new dataset.
The Algorithm Selection Problem denotes the lask of finding algorithm selector a D - A from the space of algorithm selectors S, which selects the presumably most suitable (machine learning) algorithm A' E A for a given dataset D E D where D is a set of datasets, also called mota-datasot. Here, suitability is expressed through a costly-to-evaluate performance metric m
+ R, such as accuracy or cross-entropy on the data, quantifying how well the given algorithm performs on the given dataset at the end of its training process. The (hypothetical) best algorithm selector, also called the oracle, s*, chooses the best algorithm for every dataset D on the final performances:
s* s*) = A" E arg max (D,,
(1)
AEA
Classical Algorithm Selectors sock to meta-lcarn which algorithm will perform best on a new dataset. Since the performance of algorithm A E A is costly evaluate, complete enumeration over A to select. the best one is often not feasible in practice, which leads to the application of machine learning to predict the best algorithm. To allow learning such a selector based on meta-training data, datasets D are characterized by their dataset meta-features ΦD and are fed as input to the selector. In the classical setting, these are quickly-computable properties of a dataset, called dataset meta-features Vanschoren (2018) such as, e.g., the number of data points, number of explanatory features or the entropy of a dataset's dependent variable.
We the myopia synonym for short-sightedness to highlight the limited information horizon. In particular, these kinds of predictors/selector hase their extrapolation only the observed part of and ignorant knollee....dge.
3

---

Published in Transactions on Machine Learning Research (3/2023)
For the purpose of analyzing selector's associated cost, computing ΦD is part of the selector's computational budget on a meta-test dataset to gain information about it. Optionally, can also take algorithm meta-feature Φ as input, e.g., the number of neurons in a particular layer of neural net (Tornede et al. 2020 Pulatov
ct al. 2022
In practice, many AS approaches, either implicitly (Xu al. 2012 Amadini et al. 2014) or explicitly
(Cunaa et al. 2018 Hanselle et al., 2020), compute a ranking over the algorithus and return the best one.
also rely rankings for the remainder of this paper assume that selector returns ranking from the ranking space R(A) over the discrete set A, i.e., we consider algorithm selectors of the form s D R(A..
Correspondingly,
In the classical setting, the data used to train a selector is called meta-training sel to differentiate it from the actual datasets D, on which the individual algorithms are trained and evaluated. It defined as
mmeto-train főn, &A, mD, E Dtroin A A]:
(2)
where Diroin is a set of datasets to meta-train on. Dirain is used to train a selector & to prodict rankings over the known set of algorithuns across datasets with minimal meta-train loss L (R(A) x R(A) - It
measures the error between the predicted ranking s(D) and the ground ranking (A) R(A). Hence, solve the optimization problem defined
min SES DEDirasn
Σ
((D)(A),s(D))
(3)
Similarly, the meta-test loss L' of s is measured on the known set A on hold-out datasets D E Diest
Clest (s)
Σ ((R)
(4)
DED
While L usually is differentiable training loss, during test time L' does not need to have this restriction. For the remainder, assumne this to be regret function. In particular, since algorithm selectors return recommendation of the most promising algorithms on the given datasct, we use the top-k regret for the lest-loss as defined in Eq. (4)
min (DD,A+(D))m(() Cteat (R,(A) s s(D)) AEtop-k(<(DI)
(5)
where the top-k operator returns the first & elements of the ranking that recommended and (D) =
arg max AeA m(D, A). The selector s is allowod to access the available tost-time information fØD, &a E A} for its prediction on D, resulting in the meta-test knowledge: Qmett-test D = {Φp, ea A E A).
A particularly strong kind of meta-feature classical AS is landmarking (Pfahringer et al.) 2000). Landmarking features report the validation performances m(D, A) of cheap algorithms that are usually not in the pool of candidates (A & A) as dataset meta-features. This relaxes the assumption that meta-features need to be very fast to compute.
The utility of classical and landmarking meta-features w.r.t. the selection task depends on the aligument of their description of the topology to the topology that the candidate algorithms' inductive biases are successful on as observed terms of their final test performance. This is particularly true for landmarking features as they apply their inductive biases the datasets' topology. Their relevance is related to the overlap in the functions that both sets of algorithms describe. For example, shallow decision tree might be partially informative regarding the performance of other tree-based algorithms, such random forests (Breimann, 2001) or xgboost (Chen & Guestrin 2016). We refer to the question of information content as the alignment problem of meta-features and a algorithm performance.
We refer to this meta-training set as knowlege interchangeably.
4

---

Published in Transactions on Machine Learning Research (3/2023)
Multi-Fidelity Optimization Since training machine learning model to completion can be very expensive, multi-fidelity optimization (Li et al. 2018) aims at using cheap-to-evaluate proxies to make efficient decisions, e.g. which algorithm performs best. In automated machine learning (AutoML, Hutter et al. ), these proxies can include training for a limited number of epochs or training on a subset of the training data.
Formally, we seek to approximate a costly-to-evaluate function F by a cheaper fidelity Si E J from the space of fidelities F. We note that also F part of F. Each fidelity fi comes with cost c(æ, ft) evaluate f((x)) at point A common assumption is that the approximation quality of fa w.r.t. F gets better in proportion f(), leading to trade-off between approximation quality and cost of the fidelity.
to its cost ft,,,),
Learning Curves form an ordered sub-case in multi-fidelle optimization, that is F is cost-ordered. In our case, we consider learning curve of an algorithm to be the performance metric mi D Ax R observed on cost-ordered fidelities f1...., FA. where c(x, FA) S c(2, F) and FA is the maximally observed fidelity for algorithm A. Notice, that the definition of mi generalizes the former notion of m to be observable at multiple fidelities. Formally, a learning curve can be defined as
LCD.A,FA (mc(DD,A,f))=
(6)
Notably, we distinguish between a partial learning curve, i.e,, FA + F, and a full learning curve, i.e., Fa = F. Each algorithm may be observed a different maximum fidelity FA, since a user may decide to spend different amounts of computing resources to evaluate different. algorithms A.
3 MASIF: Interpreting Partial Learning Curves Jointly
In this section, we introduce our core contributions: (i) extending the classical AS setup by adding multi- fidelity information, allowing for a more detailed analysis of the budget-regret trade-off of algorithm selectors; (ii) our transformer-bast MASIF approach as an efficient method for this extended data setup; and (iii) a data augmentation scheme to achiove strong predictive performance and enable MASIF to accept arbitrary budget allocation strategies.
3.1 Extending Algorithm Selection by Multi-Fidelity Information
In addition to datasct feta-fea we propose to use information from learning curves as multi-fidelity information for AS. This provides new source of information by prematurely inquiring about new dataset's topology from the perspective of the candidates' inductive biases that is both relatively cheap and surely well-aligned. We summarize this setup in Figure 1
Meta-Training Data The aforementioned idea extends our available training meta-knowledge in Eq. 2 and Eq. (3) by fidelity information, amounting to
mella-tre train (6n, CA, LCD.A,F E Dtrain A E A}
where the learning curves LCD.A.F E RF replace the performance values m(D,A) E R in Eq. 2 For our experiments, we observed full learning curves up to the maximal fidelity for the meta-training dataset. The meta-training dataset encodes several aspects of the available meta-knowledge: (i) the relation of the algorithm's performance w.r.t. the dataset's topology, (ii) how information about the algorithm's 's inductive bias unrolled across fidelities marginally, and (iii) how the learning behavior of different algorithms relate to each other on the datascts they applicd to.
Meta-Testing Data The meta-testing phase provides partial learning curves observed on a new hold-out dataset. The set of curves provides evidence of the dataset's topology observed through the lens of the algorithms. The available information at test time for our selector s in Eq. [4p is:
5

---

Published in Transactions on Machine Learning Research (3/2023)
Qeeta-test P FA S.AAA,
(8)
Notably, depending a user's budgetary preferences, some of the learning curves may be revealed nearly entirely or not at all; in general, we assume that the set of all learning curves may be incomplete. Since an algorithm selector returns ranking over the algorithms on the given datasct, we use the top-k regret for the test-loss on the final fidelity F as defined in Eq. (4)-
3.2 MASIF Architecture
We choose a deep learning architecture because of its functional flexibility and its non-parametric modelling. However, we have to exploit the little data typically available in AS without grossly over-fitting to the meta-data, Our proposed architecture is displayed in Figure and consists of five main building blocks:
Block 1: Dataset meta-feature embedding Assuming some valuable information on the topology in the classical meta-featuree ures (D, an MLP encodes them into a latent space. Utilizing this cheap set of features, from an informational standpoint, our method can only improve over classical AS methods and further allows us to contextualize the observed learning curves to the observed dataset. If, for instance, dD conveys some information regarding the complexity of the dataset, this may be indicative of how fast an algorithm might learn, affecting the shape of the curve.
Block 2: Learning curve preprocessing As preprocessing of the learning curves, we apply zero-padding to fill the learning curves up to the final fidelity F and accordingly generate mask to store the padding information. To avoid technical issues with fully masked sequences, we append an extra learnable End Of Sequence (EOS) token (Dosovitskiy et al.) 2020). We then apply a shared MLP separately to each position of each learning curve. Embedding each position of a learning curve individually increases the expressivity of our one-dimensional learning curve sequences into e dimensions akin to BERT's vector representations for each word (Devlin et al.} 2019). To overcome the transformer's native positional ignorance, we follow Vaswani al. (2017) in encoding the positional information with sine and cosine functions and adding these values to the embedded learning curves. This is crucial step in preserving the sequence information, which in our case conveys the order in learning curves.
Block 3: Learning curve embedding We variation of the transformer-oncodee proposed by Vaswani et al. (2017) on the set of partially observed learning curves LCD.A,F, of each algorithm A E A. This translates the (incomplete/mase) masked) learning curve sequence into a latent vector representation enriched with the meta-knowlodge of the algorithms' marginal learning behaviors.
Since the classical dataset meta-features may carry valuable information (if they are available) for the amount. and variability of information at each position in the sequence, we introduce a mechanism we dub guided attention. Similar to the classical attention module, our encoder receives query Q, key K, and value V tensor, containing the preprocessed learning curves from Block 2. We multiply the query Q of the attention heads element-wist with linear projection of the embedding described in Block 1 if dataset meta-features are available. The projection's 's role is merely to fit the shape of Q. To increase parallelism and reduce the number of learnable parameters, we use a batch trick by only processing one dataset at a time, but conceive the set of algorithms A to be a batch of dimensions [A,F, that processed in parallel. The attention-guided transformer-encoderr layer repeated N times to yicld a marginal summary of the latent representation of each of the learning curves. The order of the batch is held fixed over all datasets, s.t. the subsequent module receives the latent vector representation of algorithm's ''s learning curve at the exact same position every time.
Given the generated feature maps, we reduce the [A,F, el-dimensional lensor along the fidelity dimension, using a learnable weighted average over the fidelities F applied independently over A, resulting in e-dimensional vectors. This formulates a joint representation of the evidence on D encoded in these vectors. Given the properties of the target dataset, the model should focus on different stages of the learning curves. Therefore, make these weights learnable. More precisely, if dataset teetafeetuues are available, these weights are
6

---

Published in Transactions on Machine Learning Research (3/2023)
computed by an MLP that projects the dataset meta-feature embedding to the size of F. Otherwise, these weights directly become learnable parameters.
Block 4: Fusing the evidence. The model up until this point is still unaware of the cross-algorithm information between different learning curves, which is essential to predict the relative ranking of cach algo- rithm. Therefore, we build another transformer on top of the first transformer that fuses the evidence on the topology derived from the partial learning curves of all A E A. The rationale is that since the algorithms are always presented in the same position, the position of these vectors holds mcaning. From the perspective of BERT, each algorithm vector is the embedding to token, and the sequence of algorithms is a sentence. Therefore, we transpose the set of representations and make the batch dimension explicit A, e]. To let the transformer identify different. algorithms, we encode the algorithms' features (i.e., their hyperparameterss simple one-hot encoding when no hyperparameter is available) as positional encoding and attach them to the generated transformations. The rationale for treating the algorithm meta-feature embedding as a positional encoding - if corresponding features are available is to give the learning curve embeddings more context at best and provide an alternative index that replaces the one-hot encoding at worst. Treating these algorithm representations positional encod- ings should aid the second transformer encoder in how to fuse the sources of information. We have added an ablation on this design choice in Appendix
Soore
Decoder Layers
FM:
Concatenate
FMM
Blocky
Algo
Transformer Encoder
Algo Feature
Embedding
Blocks
Weights
Weighted Sum & Norm
Encoding
Leurning Curve Transformer Eucoder
Guided Arteution
Block
Positional
Concatenate
Encoding
FOS Embedding
Meta Frature Embedding
Input Embedding
L.C
Padding
EOS
Blocks
LC2
Padding
EOS
Blockk
Figure MASIF Architecture. Blue rectangles rep- resent data, yellow rectangles denote MLP layers, purple rectangles indicate positional encodings, and orange rectangles are transformer layers. MASIF receives the partial learning curves of all the algo- rithms LCD,A,F, AA E A (in this figure, we only show two learning curves, i.e., LC₁ and LC2, indi- cating LCD.A,,FA, and LCD,As,FA,). The two trans- former encoder layers apply attention operations the fidelity and the algorithm dimensions, respectively. Therefore, the output of each transformer needs to be transposed to fit the dimension. The output of the last transformer (FM1 and FM2) is concatenated with the embedded meta-features ΦD- Finally, the joint, features are fed to the decoder to get the predicted ranks.
Block 5: Ranking Yet again, employ an MLP the concalenation of the vectorized evidence and the dataset meta-feature embedding to fuse their in- formation and project to the size n. = IA]. The re- turned values are scores for the respective algorithms, corresponding to latent utility values, which are used to construct a ranking across the algorithms. As an objective, would like to maximize the Spearman rank correlation between predicted and true algorithm ranks. In order to obtain a differentiable loss, we thus leverage a differentiable sorting algorithm (Blondel et al. 2020 to compute the training loss.
3.3 Data Augmentation
Since AS usually only has a limited amount of dataset-algorithm combinations in its meta-training datasct for which we observe a single learning curve instantiation, respectively, we use data augmentation strategy. We mask the learning curves by sampling the available learning curve length FA uniformly and independently from [0, F), where 0 indicates a complete lack of information on that learning curve. addition to circumventing overfitier,,,,,,, this strategy allows our model to be presented with any budget allocation (combination of fidelities), i.e. (Fa E F (O))AEA including the option of not evaluating curve during test-time.
7

---

Published in Transactions on Machine Learning Research (3/2023)
3.4 Hyperparameters
To foster reproducibility, we list the following hyperparameter choices in the above architecture. Details regarding their choices, including ablations, can be found in Appendix [A.4]] We encode all the meta-features (i.c., ΦD and ΦA) with a 2-layer MLP (with hidden size of 128 and 64) into an embodding of size 64. All transformer encoder layers in MASIF have the same architecture with hidden size 128 and altention heads. Each of the transformer encoders (Learning Curve Transformer Encoder and Algo Transformer Encoder) have 2 transformer layers are appliod with dropout rate of 0.2. We train MASIF with an ADAM optimizer with learning rate of 0.001 and beta values 0.9 and 0.999 for 500 epochs while neither learning rate scheduler nor weight decay are employed.
3.5 MASIF's Benefits
The transformer encoding of the learning curves offers significant advantages: at no point do the learning curves require extrapolation. Instead, only their informational content is summarized and transformed into a conditioned ranking score. Our method, therefore, does not fall victim to error propagation as often encountered in time series regression or multi-label classification (Senge et al. 2012). Additionally, leveraging the meta-knowledge obtained by observing the learning curves of the same algorithms on meta-training datasets, we combat multi-fidelit's inherent myopia problem effectively and can be fully non-parametric in the shape of the learning curves due to strong meta-prion encoded in the weights of our architecture. To this end, MASIF exploits the available meta-knowledge extensively of having observed the algorithms' learning curves on multiple datasels jointly. Depending on the lype of algorithms and fidelity, this meta-knowledge in the form of algorithms' learning curves comes at almost no cost when collecting the meta- veeta-trannin data in the classical AS setting. Crucially, taking a multi-fidelity lens independent of whether or not we include classical and landmarking meta-features its ranking ability is no longer bounded by the alignment of tediously handcrafted and highly task-specific meta-features. Instead, it is now governed by the trade-off between the approximation quality of the fidelities and the induced cost of querying them. It does, however, readily consider expert knowledge in form of dataset meta-features to contextualize the observed learning curves.
Given some budget allocation, the MASIF model corresponds to an algorithm selector since it produces a ranking that takes all the available meta-knowledge and the gathered evidence on the test dataset into account. However, since any budget allocation is possible, it can be readily utilized not only as a selector but as an interpreter of the incrementally arriving performance feedback from the budget allocation. Queried continuously more evidence arrives, users can observe the shift in expectation conditioned what has been observed so far. This application mode makes this model an actionable support system for data science practitioners facing the AS problem.
the final ranking,
The tradc-off between approximation quality and budget costs opens up the opportunity of defining schodulers that successively queries the fidelities of algorithms (i.e., extending their learning curves) by exploiting our model. Since our contribution focuses on the model part and since MASTF's model works with arbitrary strategies for deciding the budget allocation, in our experiments we show exemplary results with the prominent scheduling approach of Successive Halving (SH) (Jamieson & Talwalkar| 2016). While the original SH procedure in each "bracket" retains the current top-performing algorithmss, our variant, dubbed SH Scheduler retains those that based on the current observation MASIF predicts to be top-performing on the final ranking.
4 Experiments
In this section, we first introduce a new evaluation protocol for our proposed extension of the classical AS meta-data setup with added fidelity information. Then, we detail our baselines along with their perks and shortcomings, briefly describe the meta-datasets we use as benchmarks and subsequently detail our findings.
8

---

Published in Transactions on Machine Learning Research (3/2023)
4.1 Fidelity-Slie Slice Evaluation Protocol
In order to assess the ability of methods to improve performance as function of the length of the observed learning curves, we introduce a standardized fidelity-slice evaluation protocol that demonstrates how a method's top-k regret w.r.t. the true ranking changes as morc and more fidclity information becomes available. This way, we can observe the progression of an information-bounde cost incurred. Tn particular, this protocol computes the regret of s())ss prediction on the test set at a gradually increasing amount of available fidelity slices; i.e. all algorithms are observed up to the same fidelity.
expectation traded with the additional
4.2 Baselines
We consider several baselines to highlight different aspects of our setup and architecture ranging from simple random rankings over classical AS to an extended version of successive halving.
Random Bascline. Consider selector that randomly guesses the ranking. It will lead to regret distribution depending on (i) & in top-k (ti) the spread and clustering of learning curves on single dataset, indicating the hardness of the ranking task and (iii) the performance scale. This baseline averages the obtainable regret of each dataset instance. It contextualizes the selectors' regrets across tasks.
Classical Algorithm Selection. While AS allows for generalization over dataset instances, its limitation in ranking performance lies in the limited expressivity and relevance of the employed meta-features. Therefore, we seek to demonstrate the benefit of partial learning curves as an addition to classical meta-features. For that purpose, we compare against SATzilla'11 (Xu al. 2012); a portfolio-based algorithm selector, which models the AS problem as multi-class classification problem and solves it using cost-sensitive all-pairs decomposition to single label classification by employing one random forest classifier for each subproblem. The final ranking is then obtained by voting. This selector is irrespective of fidelity and will produce a constant value in the slice evaluation protocol.
Fidelity enabled Algorithm Selection. Classical AS only considers the dataset and algorithms' meta- features. Considering that our method's learning curve embedding could be conceived to be latent meta-feature ΦD, classical AS can conceptually similarly be enabled. To achieve that, the classical ΦD can be extended by m(D,A, A, f) for some intermediate fidelities f < F observed for all A A. Given f E F with F' C C,, this implies that |A| IF' columns are added to the feature space. Consequently, we limit [F]| to a subset of evenly spaced fidelities In the slice evaluation protocol, the intermediate fidelities are added only up until the horizon of the protocol.
Parametric Learning Curve Predictors. Neglecting any meta-knowledge and combating myopia through strong parametric assumptions, naive learning curve predictor can interpret any partial information available. Such a predictor fits a parametric curve to each partial curve and extrapolates to the final performance. The selector s independently extrapolates to the final performances and ranks the algorithms according to these. To create a meaningful and more expressive selector that mimics a practitioner's attempt at extrapolating learning curves for AS, we fit all parametric curves described in Mohr et al. (2022) and select the best fitting one on the observed part for extrapolation. This baseline highlights that learning curve predictors are used as algorithm selectors by practitioners when lacking meta-knowledge. Surpassing this baseline indicates that our method's expectation of the ranking can benefit from joined meta-knowledge about an algorithm's 's past. progressions and related learning behaviors.
LCNet. Learning curve prediction with Bayesian neural networks Klein al. (2077b), originally designed for hyperparameter optimization problems can be applied to algorithm selection task whenever hyperpa- rameters or algorithm meta-features are available on benchmark. The underlying Bayesian Neural Network takes the hyperparameter configuration and the fidelity level at which it is observed as input and outputs both a variance and a mcan prediction. The mean is built of a weighted ensemble of the parametrizations for a fixed set of parametric learning curves. It is a meta-agnostic but expressive parametric baseline.
In our experiment, this amounts the fidelity sequence [0, 0.2, 0.4, 0.6, 0.8, 1].
9

---

Published in Transactions on Machine Learning Research (3/2023)
IMFAS. Implicit Multi-Fidelity Algorithm Selection (IMFAS) (Mohan et al. 2022 utilizes an LSTM- based architecture that initializes its hidden state using an MTP encoding of the dataset meta- veta-featuurs and auto-regressively accepts slices of fidelity to refine the expectation of the ranking.
Successive Halving (SH). Despite its simplicity, SH (Karnin et al., 2013 Jamieson Talwalkar 2016
has several desirable properties to compare against in terms of its interpretation of partial learning curves. It is a meta-agnostic and therefore myopic, fidelity-aware but non-parametric method. SH naturally produces myopic ranking, by the level at which it terminates the algorithm. Ties within this set. of terminated algorithms are resolved by their relative performances on that bracket. Limiting the fidelity information horizon of the selectors implies that for every such horizon, we need to recompute the ranking induced by SH. Notably, to permit SH the same information horizon, the last available fidelity in its schedule will be the maximum fidelity that horizon. The benefit of this convention is that SH has access to the same fidelity information and can make better decisions for the algorithms it recommends. Its drawback of SH as a baseline is that the incurred cost. differs from that of our method.
4.3 Datasets
Where possible, we utilize training, validation, and test learning curves for the training of all approaches. These are obtained by splitting each dataset in the corresponding benchmark in training, validation, and test data, training the corresponding algorithms on the training data under each fidelity, and computing their validation and test performance corresponding to the model trained for that fidelity. While the classical AS approaches only receive the final test performance on the meta-train datasets as training data, we provide (partial) validation curves to SH, the learning curve approaches, and MASIF to train on, but measure the final regret with respect to the full fidelity test performance. The same applics when the approaches are applied to new lest datasets. We then assess the performance of each approach by performing 10-fold outer cross- vals-valiiation of the meta-dataset. We use three different benchmarks (more details in Appendix A.1
Synthetic. We constructed the Synthetic meta-dataset based on parametric learning curves taken from Mohr et al. (2022), specifically to demonstrate the myopia of SH and learning curve predictors. Particularly, it introduces noisy curves, that exhibit crossing points. We show how MASIF alleviates myopia through strong prior knowledge w.r.t. the functional family an algorithm is adhering to.
Task-set. Task-sct (Metz et al. 2020) is fairly noisy real-world dataset based on paramcterizations of the Adam optimizer (Kingma & Ba 2015), , with a variety of learning rates, run on a large variety of modern deep learning architectures and datasets.
Scikit-CC18. With Scikit-CC18, we evaluated multiple well-known scikit-leard (Pedregosa et al., 2011) algorithms on the classification benchmark OpenMI.-CC18 (Bischl et al. 2021b).
LCBench. LCBench (Zimmer et al. 2021) is a learning curve benchmark of different funnel-shaped neural networks and hyperparameters for tabular data. We sub-sampled the set of configurations to only include 170 combinations algorithms to be selected.
These four benchmarks differ in their availability of dataset and algorithm meta-features: while Synthetic and Task-set exhibit neither, Scikit-CCIS and LCBench exhibits both. Since Scikit-CC18 and LCBench are the only benchmarks with available dataset meta-features ΦD, we can only evaluate SATxilla and IMFAS on them. Similarly, the dataset and algorithm meta-feature can only be exploited by our architecture on these benchmarks.
4.4 Results
Summarizing our results, depicted in Figure g on benchmarks Task-sct and Synthotic, MASIF outperforms the other methods in terms of top-1 regret, and on Seikit-CC18 it performs competitively. Additional results on an NLP subset of Task-set can be found in Appendix [A.2] as they show the overall same tendencies. Since our basclines applied in the slico-ovaluation protocol on their own already reveal a few features of the meta-datasets we would like to detail their implications first. Afterwards, we discuss our experiments in light of the available meta- knowleee w.r.t. myopia and parametric assumptions.
10

---

Published in Transactions on Machine Learning Research (3/2023)
MASIF MASIF SH
SH Imfas
Augmented SatZilla11 Parametric
LCNet Random Baseline
Satzilla
Synthetic
Task Set
50
44
30
Avsilable Fidelky SclkBtCC188
Available Fibelly
LCBench
0.12
0.30
700 Rooret
0.08
0.00
0.04
0.0G
0.02
0.00
0.00
Ausllall Fidelity
Avvillable Fiderity
Figure 3: Average lest-set lop-1 regret over Len meta-dataset folds in the slice evaluation protocol. Available fidelity is expressed in the share of the target fidelity's ' budget. The standard error bands originate from five repetitions for each of the ten folds. Note: Scikit-CC18 and LCBench exhibit dataset meta-features, while Synthetic and Task-set do not. Task-set reported for the image data subset.
Benchmark Difficulty. First of all, paying close attention to the random baseline and SH already indicates the difficulty of benchmark; the random baseline be thought of as upper regret bound for any mcaningful selector. If this threshold alrcady incurs small regret as in Scikit-CC18, this implics that there is only little to gain, increasing the difficulty for a selector to improve. SH, on the other hand, pays close attention to the earlier parts of the curve. If this is already very indicative in terms of the final performance, only few crossings of the curves occur and thus decisions based on low-fidelity are good w.r.t. final performance. In this sense, Beikit-CC15 close to oracle performance after few fidelity slices in their top-3 regrets (compare with Figure A.3). SH's detrimental performance Task-set on the other hand likely is due to the high variability and crossing of the curves, indicating the difficulty of this meta-dataset. The fact that no method obtains zero regret on Task-sct at the final fidelity originates from the noisy target as indicated by the final fidelity of the parametric baseline. Moreover, parametric learning curve is (almost) guaranteed to yield oracle performance on Synthetic by construction. Inspired by Multi-LCNet (Jawed et al. 2021), which uses LCBench as its sole benchmark, we ran our experiments on LCBench well. However, we find that LCBench is not strong benchmark for our setup, due to (i) the confidence bounds overlap (ii) the very small regret of around 0.02 of most approaches and (iii) the fact that all of the meta-aware selectors do not require additional fidclity information to adjust their prior. This is additionally supported by (i) SH's strong low-fidelity performance with only little gain by fidelity and (ii) both SH's and the parametric learning curve baseline's strong final performances.
an arguably casier meta-dataset than Task-sct, SH is
The Role of setup and expresses the prior belief of a selector in the final performance based on the knowledge Baring this in mind, the fact that MASIF almost constant and already top-performing on Synthetic indicates that the strong prior derived from meta-knowledge is already sufficient which in the case of Synthotic by construction. A decrease in regret indicates that by adding additional Lest-time fidelity information up to Fmaz refines this meta-knowledeer
We highlight the fact that zero-fidelity corresponds to the classical AS
belief.
The trajectory of the parametric learning curves is a strong myopic bascline to demonstrate the effect of mela-knowledde, as more fidelity becomes available. SH similarly acts as a myopic baseline, but its schedule's path dependency may deny to recover from early mistakes and hence result in sub-optimal states even when full fidelity is available. The steep decay in regret of MASIF at the beginning of its regret curves on Task-sct and the minor adjustments on Scikit-CCIs demonstrate the desirable property, that MASIF's prior can indeed be improved based incoming fidelity information. Similarly, IMFAS MMohan et al. 2022 exhibits,
11

---

Published in Transactions on Machine Learning Research (3/2023)
albeit far less pronounced, such decay indicating that fidelity information also helps it. Its dependence on classical dataset meta-features as initialization and its autoregressive nature prevent it from excelling at increasingly revealed fidelity information and likely even is the cause of its slight deterioration on LCBench. Its strong dependence on classical dataset meta-features also prevents it from being runnable on Synthetic and Task-set.
A direct comparison of MASIF with the parametric learning curve baseline on Synthetic and Task-set in terms of reduced variability and consistent superiority already in the carly stages highlights the benefit of having meta-learned the algorithms' learning behavior to combat myopia effectively. The SH Scheduler observed on Task-sct supports this fact. When comparing SH against SH Scheduler on this noisy dataset it is apparent, that the gained meta-knowledge ledge becomes the main driver of the regret curve. Notably, the SH Scheduler's curve appears inferior at first to MASIF's trajectory, until we factor in that MASIF follows the slice evaluation protocol very closely and the SH Scheduler has considerably less overall budget available due to the SH's budget allocation strategy. It also suffers from its path dependency, yielding slightly inferior results to MASIF, but. at considerably less cost. This regret gap diminishes with lower benchmark difficulty in terms of accuracy of the meta- keota-nnoweegg, prior beliefs facilitate making good decisions carly on. This most prominently demonstrates the budget-regret trade-off a scheduler incurs. A comparison of SATzillall and our variants on Cci8 similarly highlights this trade-off; SATzilla11 1 incurs zero additional cost at test. time over the computation of dataset meta-feature which are available to all three. Its prediction must fully rely on its prior, but MASIF can reduce SATzillall's regret at a small additional cost during test time, having learned the algorithms' learning behaviors.
can be observed on Synthetic and Scikit-CC18. when strong
To our surprise, fidelity-enabled SATzilla11 performs similarly to that without fidelity information on Scikit- CC18. This fidelity-aware but non-parametric baseline ignores the naturally occurring sequential nature and fails to improve over its fidelity-agnostic variant. It remains unclear as to why this occurs in our experiments, but we conjecture that it grossly overfits the meta- training data due to its sizable feature space.
As the ablation results in Appendix A.4 suggest, the usefulness of the raw dataset & algorithm meta-features in the two datasets is negligible at best. This has two implications: first, the meta-learned multi-fidelity information is the definite driver of our model and well outperforms the classical meta-feature based approaches. A small caveat to the results of classical methods is their handcrafted nature. Considerable time investment into the generation and preprocessing of lask-specific mela-features may yield improved regret performances for all models accepting dataset meta-features. This is why, despite the lack of strong evidence for its improvement, we keep the dataset- and algorithm meta-features as an optional part of our architecture.
5 Related Work
Classical AS approaches are based on the assumption that dataset instances (tasks) can be represented using meta-features. They learn mappings bet ween datasets and algorithms w.r.t. performance based on such pre-computed meta-features. Most methods to learn such mappings in the literature either leverage regression
techniques (Xu et al.) 2008 Bischl et al. 2016), ranking techniques (Cunha et al. 2018 Abdulrahman et al.
2018), collaborative filtering
2010 Fusi et al. 2018), instance-based learning approaches (Amadini 2010) or mixtures of the aforementioned techniques (Hanselle et al. 2020 Fehring
(Stern et al.
le al.
2014: Kadioglu et al.
et al. 2022) All of these approaches, however, suffer from the same limitation: Pre-computed meta-features
as mentioned earlier, can be in sufficient in characterizing the datasets and algorithms due to the alignment problem discussed Section [ Our approach circumvents the alignment issue by landmarking datasets directly based on the candidate algorithms' inductive biascs, as unrolled in their partial learning curves, and predicts a rank over the set of candidate algorithms.
The idea of exploiting partial learning curves has previously been explored in some works. SAM (Lette & Brazdil 2004) non-parametrically matched partial learning curves to the closest in terms of shape from the observed meta-learning curves using kNN, while van Rijn et al. (2015) reduced the cost of cross-validation by exploiting the similarity of the partially-observe rankings with those of the meta-datasets and using the most similar learning curve as surrogates. [Mohr & van Rijn| (2021) extended this approach by terminating less promising candidates early on based on their prodicted learning curves modelled in a cemmmearaettric way under the assumption of concavity. In contrast to this explicit prediction of the missing parts of the learning
12

---

Published in Transactions on Machine Learning Research (3/2023)
curves, MASIF implicitly meta-learns correlations between the partial performances of algorithms to predict a ranking. Leite & Brazdil (2010) pursue a cheaper and meta-informed alternative to cross-validation that evaluates the collection of fidelities using SAM, and then actively schedules them in a cost-aware manner.
2021a) to Similarly, multi-fidelityy optimization can be used in hyperparameter optimization (Bischl et al. save computing resources. It allows the optimizer to actively query new candidate configurations, unlike the classical AS setup. Corresponding methods can be categorized into bandit-based ones (e.g. SH (Jamieson & Talwalkar 2016), ASHA (Li et al. 2020). PASHA (Bohdal et al.] 2023), HB (Li et al., 2018 DEHB (Awad et al.
2021h, BOHB (Falkner et al. 2018), ID-HB (Brandt et al. 2023) SMAC3 (Lindauer et al. 2022)
DSMAC (Sui & Yu. 2020). BOCA (Kandasamy et al. (017)), cost-aware ones (e.g. BOIL. (Nguyen et al,
2020
FaBOLAS (Klein et al.) )017a)), and learning curve extrapolating ones (e.g. Freeze-Thess (Swersky
et al. ct al.
2014), Speeding up Automatic Hyperparameter Optimization of Deep Neural Networks (Domhan
2015). DyHPO (Wistuba et al. 2022). L.CNet (Klein et al. 2017e), as well as Multi-L.CNet (Jawed
ct al.
2021))
As Multi-LCNet is the one closest related to ours in terms of its setting on the hyperparameter optimization side, it is worth investigating the differences more closely. While they use a related (mmeta-train and @ data setup, the work has substantial differences and rigid assumptions, which make us abstain from an experimental comparison of MASIF to it, as a fair comparison is not possible. The core differences are threefold; 1. they consider a hyperparameter optimization fulti-fidelity problem 2. their model can assume the existence of auxiliary curves for a single algorithum, because of the narrow definition of an algorithm and 3. they use this multi-modal data in an autoregressive way for the explicit extrapolation of a single curve at a time. A more detailed discussion is deferred to Appendix A.5
In summary, all of these methods are applications of mulli-fidelity in hyperparameter optimization. From the lens of our setup, they are applied to a relaxation of the hyperparameter search space that is out of scope here 5 Crucially the first two kinds of methods primarily focus on the problem of scheduling the configuration and endowing it with a budget. On the other hand, the last kind of method focuses on marginally extrapolating
learning curves for single configuration irrespective of previously seen learning curves on the same dataset. MASIF differs from these approaches in that it focuses on jointly interpreting the learning curves in the AS sctup and does not imply any schedule.
Two approaches orthogonal to ours, although bearing some resemblances in terms of setup, are REVE-R (Nguyen et al. 2021) and MetaBu (Rakotoarison et al. 2022). Meta-REVEAL focuses on scheduling the algorithms and fidelities through a Reinforcement Learning perspective by modelling it as a REVEAL game. The agent acts on a discrete action space that does not account for the correlations between learning curves. As we do not focus on schedule but on learning cross-corrolations on the learning behavior, we refrain from a comparison. Consequently, using the estimates produced by MASIF can provide a pre-processed action space for their problem. MetaBu extends the classical AS idea by relating dataset meta-feature and algorithms' hyperparameters using a learned optimal transport map. They, however, ignore that the dataset meta-features cannot sufficiently characterize the datasct, thus, subsequently falling victim to the same fallacies as the methods mentioned before.
6 Conclusion
In this work, we revisited classical mota-learning- and learning curve-based approaches to AS from the perspective a trade-off between the budget they invest in inquiring information about a new dataset and the corresponding regret of choosing an inferior algorithm. Doing so, we argued that both of these selector classes are on opposite ends of the spectrum in this trade-off and both suffer from severe limitations, likely leaving data scientists with a selected algorithm that is inferior. To alleviate this, we present MASIF, an algorithm selector designed to support data scientists in the selection process in a far more native manner. Tts position in the spectrum can be controlled by the data scientist depending on their budgetary constraints. MASIF based architecture to model the performance of algorithms across both,
leverages a fidelities and algorithms in the form of learning curves of varying lengths. As such, it alleviates the myopia
USH-based hodh are living on discretized version of this broader space.
13

---

Published in Transactions on Machine Learning Research (3/2023)
of many existing learning curve-based approaches while making parametric assumptions the learning curves it can model. Crucially, our selector utilizes computation from previous selections as meta-knowledge In an extensive experimental study on four different benchmarks, we showed that MASIF outperforms existing notta-learning-base based approaches in terms of the regret of the selected algorithm and learning curve-based algorithm selectors lerms of regret. for the invested budget. As such, MASIF not only an AS approach designed to support the data scientist in practice by leaving the concrete instantiation of the trade-off their preference but also yields state-of-the-art AS performance.
7
Future Work
The aforementioned budget-regret trade-off leads to multi-objective view of the AS problem, where both the budget invested and the regret of the corresponding algorithm selector rivalling objectíves. This holistic view naturally suggests tackling the problem with multi-objective methods to be able to present Pareto front of selectors to choose from, suited to a data scientist's 's budget-regret preferences in the face of uncertainty. This line of work is orthogonal to previous multi-objective work in Algorithm Selection Bossck & Trautmann 2012 Any instance on this Pareto front a scheduler of sorts. Determining how a scheduler knowledgeable in learning behaviors should choose its budget allocation sequentially and in the face of uncertainty inspires research on its own. Lastly, in the current form of this work, the algorithms are predominantly represented through their learning curves and limited manner. Multi-LCNet (Jawed et al. 2021 presumes that using hyperparameters among other informative algorithm mela-fealures may also be beneficial in contexlualixing a learning curve. In that sense, natural extension to the scope of this work is the transition from tackling the AS problem to tackling the hyperparameter optimization problem. In contrast to cxisting approaches, we will seok to extract the joint topology evidence derived from the observed partial learning curves order to lackle this problem.
algorithm meta-features in relatively
Acknowledgements
Marius Lindaner and Tim Ruhkopf acknowledge financial support from the Federal Ministry for Economic Affairs and Energy of Germany in the project CoyPu under Grant No. 01MK21007L.
Frank Hutter acknowledges financial support by the European Union (via ERC Consolidator Grant "DeepLearn- ing 2.0", grant no. 101045765). Alexander Tornede acknowledges similar financial support by the European Union (via ERC, "ixAuloML", grant no.101041029). Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
The authors gratefully acknowledge the computing time provided to them on the sigh-performance computers Noctua2 at the NHR Center PC2 under the project hpe-prf-intexml. These are funded by the Federal Ministry of Education and Research and the state governments participating on the basis of the resolutions of the GWK for the national high performance computing at universities (www
dew/dd/////neee /uuuuuunssrraaaeee paastrr..r..er.....
erc
European Research Council
14

---

Published in Transactions on Machine Learning Research (3/2023)
References
S. Abdulrahman, P. Brazdil, J. van Rijn, and J. Vanschoren. Speeding up algorithm selection using average ranking and active testing by introducing runtime. Machine Learning, 107(1):79 108, 2018.
R. Amadini, M. Gabbrielli, and J. Mauro. SUNNY: lazy portfolio approach for constraint solving. Theory and Practice of Logical Programming, 14(4-5):509-624, 2014.
Anaconda. The state of data science 2020: Moving from hype toward maturity. Anaconda, 2020.
N. Awad, N. Mallik, and F. Hutter. DEHB: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Z. Zhou (ed.), Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI'21, pp.
ijeai.org, 2021.
B. Baker, O. Gupta, R. Raskar, and N. Naik. Accelerating neural architecture search using performance prediction. In International Conference on Learning Representations Workshop track, 2018. Published online: iclr.cc
B. Bischl, P. Kerschke, L. Kotthoff, M. Lindauer, Y. Malitsky, A. Frechétte, H. Hoos, F. Hutter, K. Leyton- Brown, K. Tierney, and J. Vanschoren. ASlib: A benchmark library for algorithm selection. Artificial Intelligence, 237:41 58, 2016.
B. Bischl, M. Binder, M. Lang, T. Pielok, J. Richter, S. Coors, J. Thomas, T. Ullmann, M. Becker, A. Boulesteix, D. Deng, and M. Lindauer. Hyperparameter optimization: Foundations, algorithms, best practices and open challenges. arXiv:2107.0588 2021a.
B. Bischl, G. Casalicchio, M. Feurer, F. Hutter, M. Lang, R. Mantovani, J. van Rijn, and J. Vanschoren. Openml benchmarking suites. In J. Vanschoren, S. Yeung, and M. Xenochristou (eds.), Proceedings of the Neural Information Processing Systems Track on Dalasets and Benchmarks. Curran Associates, 2021b.
M. Blondel, O. Teboul, Q. Berthet, and J. Djolonga. Fast differentiable sorting and ranking. In H. Daume III and A. Singh (eds.), Proceedings of the 37th International Conference on Machine Learning (ICML '20), volume 98. Proccedings of Machine Learning Research, 2020.
O. Bohdal, L. Balles, B. Ermis, C. Archambeau, and G. Zappella. PASHA: efficient HPO with progressive resource allocation. In Proceedings of the International Conference Learning Representations (ICLR '23), 2023. Published online: iclr.cc
J. Bossek and H. Trautmann. Multi-objective performance measurement: Alternatives to PAR10 and expected running time. In Y. Hamadi and M. Schoenauer (eds.), Proceedings of the Sixth International Conference on Learning and Intelligent Optimization (LION'12) volume 7219 of Lecture Notes in Computer Science, pp. 215-219. Springer, 2012.
J. Brandt, M. Wever, D. Iliadis, V. Bengs, and E. Hüllermeier. Iterative deepening hyperband. arXiv:2302.00511 2023.
L. Breimann. Random forests. Machine Learning Journal, 45:5 32, 2001.
T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In B. Krishnapuram, M. Shah A. Smola, C. Aggarwal, D. Shen, and R. Rastogi (ods.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 785 794. acm, 2016.
K. Cho, B. Merrienboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXic:1409.1220 1259, 2014.
H. Coelho, R. Studer, and M. Wooldridge (ods.). Proceedings of the Nineteenth European Conference on Artificial Intelligence (ECAI110, 2010. IOS Press.
T. Cunha, C. Soares, and A. de Carvalho. A label ranking approach for selecting rankings of collaborative filtering algorithms. In H. Haddad, R. Wainwright, and R. Chbeir (eds.), Proceedings of the 33rd Annual A ACM Symposium Applied Computing (SAC'18,, pp. 1393 1395. ACM, 2018.
15

---

Published in Transactions on Machine Learning Research (3/2023)
J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pro-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-HLT, pp. 4171-4186. Association for Computational Linguistics, 2019.
T. Domhan, J. Springenberg, and F. Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Q. Yang and M. Wooldridge (eds.), Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI'1), pp. 3460 3468, 2015.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv:1200 20000. 11929, 2020.
S. Falkner, A. Klein, and F. Hutter. BOHB: Robust and efficient hyperparameter optimization at scale. In J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning (ICML'18) volume 80, pp. 1437-1446. Proceedings of Machine Learning Research, 2018
L. Fehring, J. Hanselle, and A. Tornede. HARRIS: Hybrid ranking and regression forests for algorithm selection. In F. Ferreira, Q, Lei, E. Triantafillou, Vanschoren, and H. Yao (eds.), NewrIPS 2022 Workshop on Meta- Learning (MetaLearn 2022), 2022.
N. Fusi, R. Sheth, and M. Elibol. Probabilistic Matrix Faclorization for Automated Machine Learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cosa-Bianchi, and R. Garnett (eds.), Proceedings of the 31st International Conference Advances in Neural Information Processing Systems (NewrIPS 188) pp. 3348 -1177 Curran Associates, 2018.
J. Hanselle, A. Tornede, M. Wever, and E. Hüllermeier. Hybrid ranking and regression for algorithm selection. In U. Schmid, F. Klügl, and D. Wolter (ods.), Proceedings of the 43rd German Conference on AT (KI'20), volume 12325 of Lecture Notes in Computer Science, pp. 59 72. Springer, 2020.
F. Hutter, L. Kotthoff, and J. Vanschoren (eds.). Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019. Available for free at http:/ /at.t/aoooooo///////// /book.
J. Petrak J. Fürnkranz. An evaluation of landmarking variants. In L. Raedt and P. Flach (eds.), ECML/PKDD 2001 Workshop Integrating Aspects of Data Mining, Decision Support and Meta- Meta-Learning, pp. 57-68. Springer, 2001.
K. G. Jamieson and A. Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS, volume 51 of JMLR Workshop and Conference Proceedings, pp. 240 248. JMLR.org, 2016.
S. Jawed, H. Jomaa, L. Thieme, and Grabocka. Multi-task learning curve forecasting across hyperparameter configurations and datasets. In N. Oliver, F. Pérez- Cruz, S. Kramer, J. Read, and J. Antonio Lozano (eds.), Proceedings of the 22nd Europeon Conference on Machine Learning and Principles und Practice of Knowledge Discovery in Databases (ECML/PKDD) volume 12975 of Mochine Learning and Knowledge Discovery in Databases, pp. 485-501. ACM, 2021.
S. Kadioglu, Y. Malitsky, M. Sellmann, and K. Tierney, ISAC instanco-specific algorithm configuration. In Coelho et al. 2010), pp. 751-756 -56..
K. Kandasamy, G. Dasarathy, J. Schneider, and B. Póezos. Multi-fidelity bayesian optimisation with continuous approximations. In Precup & Teh (017).
Z. Karnin, Koren, and O. Somekh. Almost optimal exploration in multi-armed bandits. In Dasgupta and D. McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning (ICML 113, pp. 1238 1246. Omnipress, 2013.
16

---

Published in Transactions on Machine Learning Research (3/2023)
P. Kerschke, H. Hoos, F. Neumann, and H. Trautmann. Automated algorithm selection: Survey and perspectives. Buolutionary Computation, 27(1):3 -5, 2019.
D. Kingma and Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (TCIR'15) 2015. Published online: iclr.cc
A. Klein, S. Falkner, S. Bartels, P. Hennig, and F. Hutter. Fast bayesian hyperparameter optimization on large datasets. In Electronic Journal of Statistics, volume 11, pp. 4945 4968, 2017a.
A. Klein, S. Falkner, J. Springenberg, and F. Hutter. Learning curve prediction with Bayesian neural networks. In Proceedings of the International Conference on Learning Representations ([CLR'17) 2017b. Published online: cclr.cc
A. Klein, S. Falkner, J. Springenberg, and F. Hutter. Learning curve prediction with bayesian neural networks. In Precup & Teh (2017).
Leite and Brazdil, Predicting relative performance of classifiers from samples. R. Greiner (ed.), Proceedings of the 21st International Conference on Machine Learning (ICML Omnipress, 2004.
R. Leite and P. Brazdil. Active testing strategy to predict the best classification algorithm via sampling and metalearning. In Coelho et al. (2010), pp. 309-314.
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit- bastdd approach to hyperparameter optimization. Journal of Machine Learning Research, 18(18)):1-52, 2018.
L. Li, K. Jamicson, A. Rostamizadeh, E. Gonina, Ben-tzur, M. Hardt, B. Recht, and A. Talwalkar. A system for massively parallel hyperparameter tuning. In Proceedings of Machine Learning and Systems, 2020.
M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. Benjamins, T. Ruhkopf, R. Sass, and F. Hutter. SMAC3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research (JMLR) MLOSS, 23(54):1 9, 2022.
D. Long, S. Zhang, and Y. Zhang. Performance prediction based on neural architecture features. Cognétive Computation and Systems, 2(2):80-83, 2020.
L. Metz, N. Maheswaranathan, R. Sun, C. Freeman, B. Poole, and J. Sohl-Dickstein. Using a thousand optimization tasks to learn hyperparameter search strategies. arXix:12002. 11887, 2020.
A. Mohan, T. Ruhkopf, and M. Lindauer. Towards meta-learned algorithm selection using implicit fidelity information. In ICML 2022 Workshop Adaplive Experimental Design and Active Learning in the Real World (ReALML 2022), 2022.
F. Mohr and J. van Rijn. Fast and informative model selection using learning curve cross-validation. 1rXii:12211.139 13914, 2021.
F. Mohr and J. van Rijn. Learning curves for decision making in supervised machine learning A survey. arxiv:2201.12 12150, 2022.
F. Mohr, T. Viering, M. Loog, and J. van Rijn. Ledb 1.0: An extensive learning curves database for classification lasks. In M. Amini, S. Canu, A. Fischer, T. Guns, P. Novak, and G. Tsoumakas (eds.), Proceedings of the 23rd European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD), volume 13717 of Lecture Notes in Computer Science. Springer, 2022.
M. Nguyen, N. Grinsztajn, I. Guyon, and L. Sun-Hosoya. MetaREVEAL: RL-based Meta-learning from Learning Curves. In G. Krempl, V. Lemaire, D. Kottke, B. Hammer, and A. Holzinger (eds.), ECML/PKDD 2021 Workshop on Interactive Adaptive Learning, 2021.
V. Nguyen, S. Schulze, and M. Osborne. Bayesian optimization for iterative learning. 2020.
17

---

Published in Transactions on Machine Learning Research (3/2023)
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
B. Pfahringer, H. Bensusan, and C. Giraud-Carrier. Meta-loarning by landmarking various learning algo- rithms. In P. Langley (ed.), Proceedings of the Seventeenth International Conference on Machine Leurningg (ICML 100), pp. 743-750. kaufmann, 2000.
D. Precup and Y. Teh (eds.). Proceedings of the 34th International Conference Machine Learning (ICML'IV 17), volume 70, 2017. Proceedings of Machine Learning Research.
D. Pulatov, M. Anastacio, L. Kotthoff, and H. Hoos. Opening the black box: Automated software analysis for algorithm selection. In Guyon, M. Lindauer, M. van der Schaar, F. Hutter, and R. Garnett (eds.), Proceedings of the First International Conference on Automated Machine Learning, volume 188 of Proceedings of Machine Learning Research, pp. 6/1 18. PMLR, 25 27 2022.
Rakotoarison, Milijaona, A. Rasoanaivo, M. Scbag, and M. Schoenauer. Learning meta-fcatures for autoML. In Proceedings of the International Conference on Learning Representalions (ICLR 222), 2022. Published online: iclr.cc
J. Rice. The algorithm selection problem. Advances in Computers, 15:65-118, 1976.
R. Senge, J. José del Coz, and E. Hüllermeier. On the problem of error propagation in classifier chains for multi-label classification. In M. Spiliopoulou, L. Schmidt-Thieme, and R. Janning (eds.), Proceedings of the 36th Annual Conference of the Gesellschaft für Klassifikation e. V., Studies Classification, Data Analysis, and Knowledge Organization. Springer, 2012.
D. Stern, H. Samulowizz, R. Herbrich, T. Graepel, L. Pulina, and A. Tacchella. Collaborative expert portfolio management. In M. Fox and D. Poole (eds.), Proceedings of the Twenty-fourth National Conference Artificial Intelligence (AAAT'10). AAAT Press, 2010.
G. Sui and Y. Yu. Bayesian contextual bandits for hyper parameter optimization. IEEE Access, 2020.
K. Swersky, J. Snoek, and R. Adams. Frecze-thaw bayesian optimization. arXiv:11406. 3849999966. 2014.
Tornede, M. Wever, and E. Hillermeier, Extreme algorithm selection with dyadic feature representation. In A. Appice, G. Tsoumakas, Y. Manolopoulos, and S. Matwin (eds.). Proceedings of the International Conference on Discovery Science (DS), pp. 609-324. Springer, 2020.
J. van Rijn, S. Abdulrahman, P. Brazdil, and J. Vanschoren. Fast algorithm selection using learning curves. In E. Fromont, T. De Bic, and M. van Leeuwen (cds.), Proceedings of the 14th International Symposium Intelligent Data Analysis. Springer, 2015.
J. Vanschoren. Meta-learning: A survey. 11Xiv:11810.0 03544, 2018
J. Vanschoren, J. van Rijn, B. Bischl, and L. Torgo OpenML: Networked science machine learning. SIGKDD Explorations, 15(2):49 60, 2014.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Proceedings of the 30th International Conference on Advances in Newral Information Processing Systems (NeurIPS'17) Curran Associates, Inc., 2017.
M. Wistuba, A. Kadra, and J. Grabocka. Dynamic and efficient gray-box hyperparameter optimization for deep learning. ar crXii:2200..00 09774, 2022.
L. Xu, Hutter, H. Hoos, and K. Doston-Brownr SATzilla: Portfolio-basee algorithm selection for SAT. Journal of Artificial Intelligence Research, 32:565-600 6066, 2008.
18

---

Published in Transactions on Machine Learning Research (3/2023)
L. Xu, F. Hutter, J. Shen, H. Hoos, and K. Leyton-Brown. SATzilla2012: improved algorithm selection based on cost-sensitive classification models. In A. Balint, A. Belov, D. Diepold, S. Gerber, M. Järvisalo, and C. Sinz (eds.), Proceedings of SAT Challenge 2012: Solver and Benchmark Descriptions, pp. 57 58. University of Hclsinki, 2012.
L. Zimmer, M. Lindauer, and F. Hutter. Auto-PyTorch Tabular: Multi-Fidclity MetaLcarning for Efficient and Robust Au IEEE Transaclions on Pallern Analysis and Machine Intelligence, pp. 3079 3090, 2021.
19

---

Published in Transactions on Machine Learning Research (3/2023)
A Appendix
A.1 Meta-Datasets
Synthetic To demonstrate that MASIF is non- myopic approach, we create a synthetic function meta-dataset. A snapshot the synthetic curve meta-dataset can be found under Figure | the myopic algorithms, we force the curves in one dataset to intersect at least. another curve in in- crementally way. We initialize by randomly picking parametric curve. Subsequent curves are gener- ated by first sampling parametric family as de- scribed by Mohr et al. (2022) and then computing its parametrization such that is ensured inter- sect with its predecessor. In more detail, we want ensure that the intersections occur different stages of the training procedure. This is obtained by randomly selecting one of lew preset intervals in which the intersection is supposed to take place. Repeating this process for - - times produces single dataset. To make the task more challenging, we add random noises to the generated curves. To obtain strong meta-knowledge across datasets originating from the shape of an algorithm's curve, a new dataset generated by perturbing the parametrization the existing curves. Therefore, these datasets contain similar but non-identical curves, that yield variability in the final ranking. This dataset does not provide dataset meta-features. Overall, we collect 22 datasets each containing 30 learning curves with 51 fidelities.
Synchetic Curves
80
To break
60
20
0.0
Fideitty
Figure 4: snapshot of a Synthetic dataset.
Task-set Task-set Metz et al. 2020 is meta-dataset that consists of over thousand tasks ranging from image classification with fully connected convolutional neural networks to variational autoencoders variety of datasets. Each task is characterized by 1. an initialization function 2. data split 3. loss function
4. gradients
On these tasks, the training, validation, and test curves of multiple optimizer settings have been recorded. The tasks are additionally organized into families, each pertaining the kind of problems that they are trained on. For experiments, sub-sampled the tasks from 2 different categories of tasks: Image Recognition and Language Modeling on text data.
The image recognition subset comprises curves generated from fully-connected networks run only on image recognition tasks, encompassing the following families:
1. mlp: fully-comnected networks trained image data
mlp_ac: ac: MLP-based autocncoder trained on image data
3. mlp_van vac: MLP--bsed VAE trained image data
The language modelling subset comprises curves generated from Recurrent Neural Networks (RNNs) trained on text data, encompassing the following three families:
1. char rmn word rmn language_ Language modelling with an RNN on words and subwords.
Language modelling with an RNN on characters.
3. rnn text classificcct: Text classification using RNN models.
We sample the set of algorithms be the different configurations of the Adam optimizer (Kingma & Ba 2015) with variations of the learning rate. This allows us to create meta-dataset. of 1000 configurations
20

---

Published in Transactions on Machine Learning Research (3/2023)
MASIF MASIF + SH
SH imfas
Augmented SatZilla11 Parametric
LCNet Random Baseline
Satzilla11
Task Set NLP
Task Set NLP
90
80
70
60
50
Aoullble Fidelity
Available Fidelity
Figure 5: Average test-set top-1 and top-3 regret over ten meta-dataset folds in the slice evaluation protocol for the NLP subset Task-set. Available fidelity expressed in the share of the target fidelity's 's budget. The standard deviation originates from five repetitions for each fold.
100 tasks. This meta-dataset does not provide dataset meta-features The rationale is that given the noise and likely crossings of the curves sampled from realistic tasks, the performance initial fidelities is not indicative of the final ranks, making it hard for methods like Successive Halving to get good regret value without any form of meta-knowledge.
Scikit-CC18 OpenML-CCI CC18 (Bischl et al. 2021b) is curated classification benchmark featuring 72 carefully selected datasets from OpenML. (Vanschoren et al. 2014) with a variety of desirable properties for a benchmark (see [Bischl et al. 2021b) for details). We generated learning curves on these datascts for 16 classifiers from Scikit-learn (Perregosa al. 2011), namely (i) TreissTrrelllssifferr (ii) Devision/rreCllssiffe (iii) MLPClassifier (iv) KNNeighoorClaassifeer (v) SGDClassifier (vi) RidgeClassifier (vii) PassiveAggres-
siveClassifier (vii) Gradient Bootteentoosttigglaassst (ix) Extra TeeeraTreeCClsssinii (x) BernoulliN (xi) LincarSVC
(xii) LogisticRRereesionn (xiii) MultinomialNB (xiv) Nearest (xv) Perceptron (xvi) SVC.
We collect their learning curves performing the following procedure for every dataset and learner pair: We performed three-fold cross-validation and for each of these folds again split 20% off of the training data as validation data. We then trained the classifier on 5%, 10%.... 100% 100% the training data and evaluated on
the train, test, and validation data using accuracy as a loss function. As preprocessing, we imputed missing feature values with the most frequent one in the training data and one-hot encoded categorical features. Correspondingly, the budgeted resource is the dataset subset size for this meta-dataset. This dataset provides dataset meta-features.
LCBench LCBench Zimmer et al. 2021 is meta-dataset, that consists of 2000 hyperparameter configurations of funnel-shaped neural net on 35 datasets. To meet the AS setup, we choose subset of size 170 of these configurations using a top-k ensembling procedure. In particular, this procedure builds the union of algorithms over the top-10 performing algorithms per dataset across datasets to ensure that cach algorithm was least somewhat. successful on at least one dataset. Presuming, that hyperparameters change the inductive bias of model, we conceive for the purpose of our analysis these configurations as independent algorithms. Notably because of this, the hyperparameter optimization usually resorts assuming similarity of performance w.r.t. hyperparameters. By extension, expect that assume some similarity in the learning behaviors encoded in their learning curves, that MASIF should be able to exploit. This meta-datasot provides dataset meta-features.
A.2 Task-set NLP
On the NLP task, depicted in Figure 5 we see the overall tendencics of the image subset Task-set, with the exception that the parametric learning curve baseline is not able to close the performance gap provided full fidelity.
21

---

Published in Transactions on Machine Learning Research (3/2023)
MASIF MASIF
SH Imfas
Augmented SatZilla11 Parametric
LCNet
Satzilla
SH
Random Baseline
Synthetic
Task Set
50
40
2:50
20
3.40
10
Available Fidelky
Available Fibelly
Sokit-CC18
LCBench
0.030
0.025
0.08
0.020
0.006
11 00.04
0.02
0.005
0.00
Acsilable Fiielity
Availablle Fidelity
Figure 6: Average lest-set top-3 regret over Len meta-dataset folds in the slice evaluation protocol. Available fidelity is expressed in the share of the target fidelity's budget. The standard deviation originates from five repetitions for each fold. Scikit-CC18 and LCBench exhibit dataset meta- features, while Synthetic and Task-set do not. Task-set is reported for the image data subset.
MASIF
MASIF NoMD
MASIF NoMA
MASIF NoMDA
Random Baseline
Scikt-CC18
LCBench
0.150
0.06
0.100
0.04
0.03
0.000
0.025
0.0
0.2
6.4
0.8
1.0
0.0
0.2
0-4
0.6
0.8
3.0
0.015
0.04
0.00
0.0
0.2
0.5 Available Fidekky
0.2
0.8
1.0
0.00
0.0
0.2
0.6 Avaiable Rdallty
0.4
0.5
Figure 7: Ablation of MASIF w.r.t. whether or not meta-features are available Scikit-CC18 (Left) and LCBench (Right) benchmarks. 'NoM' indicates that the subsequently detailed meta-feature information is hidden to the model. In particular, 'D' stands for dataset meta- meta-features, 'A' for algorithm meta-features.
A.3 Top-3 results
As increasing k in top-k is a hedge, the overall regret is reduced compared to those results of top-1, because it is more likely to pick the best performer in this enlarged set. The overall tendencies Figure are however exactly the same as described in Section 4.4
A.4 Ablations
A.4.1
Dataset & Algorithm Meta Features
The results F indicate that the regret difference in whether or not the meta-features are present have if any only negligible effect for the Scikit-CC18 and LCBench dataset, once the scale of this difference is considered. On the other hand, the results in Figure g detail, the regret gap, when using a secondary
22

---

Published in Transactions on Machine Learning Research (3/2023)
MASIF
MASIF with MLP
Random Baseline
Task Set
Task Set
70
30
60
-50
Too-1
40
Too-
20
6.6 Ausiante hdwiiiy
0.4
Aouilable Fidelity
Figure 8: Ablation MASIF on Task-set benchmark, switching out the second transformer for a simple MLP on the reduced output as joint interpretation module. Task-set is reported for the image data subset.
transformer over the sequence of algorithms rather than an MLP, supporting our design choice for the former. The vector representation of the learning curves output to the first transformer holds positional information since an algorithm is presented always in the same order irrespective of the dataset. The secondary transformer picks up on this positional information more easily than an MLP.
A.4.2 Parametrization
Provided the scarce data available, cannot reasonably perform hyperparameter optimization the meta-level without risking overfitting. Instead, we consider how the design choices regarding the model's hyperparamcters are affecting performance marginally on Task-set, depicted in Figure g As AdamW with 5% linear warm-up and cosine decay is a default choice for optimizing transformers, we added an ablation to it. Since the number of attention heads, the number of transformer layers affect the model's capability in describing functions, these hyperparameters are relevant choices. Similarly, dropout should help the robustness of the transformer. We do, however, find that our model is relatively robust to changes in these hyperparameters. The overall tendencies and results are the same across hyperparameter configurations. The slight performance improvements for the marginal alterations prompted us towards trying their combination, i.c. AdamW with linear warm-up and cosine decay with learning rate of 0.001, dropout of 0.4 and 4 as number of transformer layers (as depicted in Figure 9 bottom right). To validate the design choice, we checked the performance of this promising combination on the remaining benchmarks as depicted in Figure 10 as a validation set. We do not find the improvements observed on Task-set to consistently yield improvements on these benchmarks, which is why we are defaulting to the hyperparametrriztion described in Section 3.4
A.5 Discussion on Multi-LCNet
As conceptually the closest to our model, we will discuss the differences summarized in more thoroughly. First, their method considers hyperparameters, which allows exploiting the similarities in that space interpolate between algorithms- which cases our assumption of a discrete and rigidly observed A in the meta-training set and therefore is rather hyperparameter optimization than AS. Second, they leverage mudti-modal data; i.e. multiple auxiliary curves such as e.g. the layer's gradient information are tracked during training. These curves are only accessible for all configurations, because they limit themselves to a single class of algorithms, in their case funnel-shaped neural nets. These secondary curves are not necessarily available for all candidate algorithms considered an AS model and as such, the method cannot be applied in our setting, Besides, considering SH's almost oracle performance on LCBench in Figure with sizeable subset of 170 algorithm configurations in experiments, the multi-modal nature of their analysis may not be driver for their performance this particular dataset. Third, their focus is explicitly extrapolating single lcarning curves at higher fidclity in an auto-regressive manner based on the meta-learned weights of their Gated Recurrent Units (Cho et al. 2014) that only combine the information from the partial learning curves of the same algorithm observed from multiple modes of data. Instead, our method leverages the joint evidence of all invested computations addition to the meta-knowledge and avoids explicit extrapolation. Fourth, their handcrafted loss metric is intended foster good predictions early on, which, despite working with rather small budgets, might yield suboptimal decisions. Similarly to the SH baseline, this seems to
23

---

Published in Transactions on Machine Learning Research (3/2023)
MASIF
Jr,0,0001
MASIF If 0.01 MASIFIICOLI
Random Baseline
MUST F-222 FORTN
MAGE 0.3 SERTI
Task Set Ablation on Learning Rate
Task Ablation on AdamW Optimizer
100-1 Regret
60
Top-1 Regret
50
40
30
20
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Available Fidelity
Available Fidelity
NUSE
MUSE 43 0 MASE
Ransen Baseline
MASF _ntflayer__
MASIF
MASIF Itflayers
Random Beseline
Task Set Ablation on DropOut
Task Set Ablation on Number of Transformer Layers
70
60
Regret
Ragret
40
Top-1
30
30
20
20
0.0
6.2
0.4 Available Fidelity
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Available Fidelity
MASIF dinddl rhead & MASIF dicde 84, /head B
Meder dmantee AiAiileeds'
MASIF
MASIF nttlayers, Ranodin Saselife
ntftlayerss4
MASIF T. 0.001 adamm
Task Set Ablation on Transformer Architecture
Task Set Ablation on Marginal Best
70
60
60
Regret
Top-1 1 Ragret
50
50
T00:1
30
20
20
0.0
6.2
0.6 Available Fidelity
0.4
0.8
1.0
0.0
0.2
0.6 Available Fidelity
0.4
0.8 1.0
Figure 9: Ablations of MASIF with marginal changes in the hyperparametrriation on Task-set benchmark on all its folds with 5 seeds respectively. The marginally changed parameters are Adam's learning rate (Ir), the learning rate of AdamW with 5% linear warm-up and cosine decay (Ir adamw), dropout (dp), number of attention heads (n head), the number of transformer layers (nftlayers) and their hidden dimensions (d model). Notably, we jointly looked at d model & heads. The default configuration is depicted in blue and amounts to 1r-0.001, dp-0.2, heads=4, utflayers-2, d model=128. Task-set is reported for the image data subset.
express and favour a particular budgetary preference over the resulting regret. In contrast, as mentioned several times, MASIF much more flexible regarding the budget-regret trade-off.
24

---

Published in Transactions on Machine Learning Research (3/2023)
MASIF
MASIF Ir 0.00110001000000 dpp 0.4, nntllyyerrr...mmm 4, a.adamw
Random Baseline
LCBench
Sclkit-CC18
0.15
0.12
0.10
Top-1 Regret
Regret
0.10
0.08
0.06
0.05
0.02
0.00
0.2
0.4
0.8
0.00
0.0
0.2
0.8
1.0
Available Fidelity
Available Fidelity
Synthetic
Regret
40
30
20
0.0
0.2
0.4
0.6
0.8
1.0
Available Fidelity
Figure 10: We apply the optimal configuration the Taskset benchmarks the other benchmarks. However, the optimal configuration on the Taskset benchmark does not show better performance compared to MASIF's default setting.
25

---

Published in Transactions on Machine Learning Research (3/2023)
A.6 Reproducibility Statement
Where can the code be found? MASIF's code is published on
What hardware did we use for the experiments? All the experiments are executed on 4 Intel Xeon E5 cores with 8000MB RAM.
What hyperparameter settings did we use? How did we get to those? Where they be found? Using Hydra (hydra.cc) cc) as base package for all our experiment pipelines including the preprocessing of dataset & algorithm meta-features as well as those for the learning curves, all of the configurations detailing the used models and their configurations can be found in the "configs" folder in the linked repository.
What are the requirements in terms of packages and version numbers? The packages & version numbers are available in the setups file of the linked repository.
Where can you download our benchmark datasets? We added the newly created Seikit-CC18 be downloaded from https://github
benchmark supplementary material. LCBench https://github: com Con//autonl/LLBench) Task-set can be obtained from ggoglearraearrdd//oooe/aasttr// set, We provide the Synthetic and Scikit-CC18 benchmarks as supplementary
How many sceds/splits etc. did perform? We used five seeds and ten folds for each benchmark. A crucial detail regarding the split in the algorithms' learning curves is described in the dataset section.
26


Reviews:
[{"review_id": "BVCtHPMhif", "reviewer": "Reviewer_7vrQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Combining both multi-fidelity information and learning curve prediction is an interesting challenge and does not seem to have been fully and satisfactorily handled in the past.\n- The paper presents a fairly natural solution to the problem, and the presentation is for the most part well-done, if sometimes a bit tedious.\n- The solution is evaluated on a range of meta-datasets.\n\nWeaknesses:\n- There are numerous seemingly related methods that could have been compared to or at least discussed. On the comparison side, successive halving is not really the leading method in multi-fidelity optimization, with more advanced variants like BOHB and ASHA being widely used. On the discussion (and perhaps comparison) side, what is the relationship between this paper and an approach like OptFormer (Chen et al., NeurIPS 2022)?\n- The proposed method only seems obviously better at low fidelity, and the improvement due to SH is inconsistent (e.g. it is worse on TaskSet), so there is no single solution that emerges.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an approach based on a meta-learned Transformer to jointly use multi-fidelity and task feature information to perform algorithm selection. It uses a Transformer trained on part of a meta-dataset of algorithmic selection tasks to train a ranking predictor, which it then evaluates in combination with successive halving on a variety of tasks. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Addressing the line comments below would strengthen the work, but are largely easy/not critical. I do believe the last point (line 346) on lack of comparison with the latest SH methods is the biggest in terms of potential greater impact of the paper. I also think it is important to at least mention the OptFormer work and compare methodology (I understand a direct empirical comparison might be difficult).\n\n34: perhaps the survey paper of Mohr & van Rijn (2022) should be linked here to summarize the field, not a single unpublished paper by the same authors?  \n43: “with few exceptions, such as (van Rijn 44 et al., 2015; Klein et al., 2017; Baker et al., 2018; Long et al., 2020)” - it seems weird to say “few exceptions” here when only two works (and by the same exact authors, Mohr & van Rijn) were pointed to as examples of the general approach.  \n100: are these rankings strict or partial orderings?  \n106: Outside a multi-objective setting, why do we care about the ranking error, e.g. why does it matter if we rank the second-worst and worst algorithms incorrectly?  \n186: unclear if a lot of discussion of positional encoding is necessary; while the Transformer model indeed ignores positional information, the solution you use is the one used in the original paper to solve the same exact problem.  \n343: what is the difference between the present paper and IMFAS, other than going from LSTM to Transformers? Could it not also be combined with SH?  \n346: a variety of strong, widely used approaches have been built on top of SH, including Hyperband, ASHA, and BOHB. Comparison with those would strengthen the paper\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "h79Yn3g4nD", "reviewer": "Reviewer_bHsm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Transfer learning for algorithm selection is an important problem that can benefit from flexible, multi-fidelity approaches.\n\nWeaknesses:\n- MASIF requires limiting algorithm configuration space to a fixed set.  This can be a significant limitation for large search spaces where it is difficult to get good coverage with a fixed set.  It also limits the applicability of MASIF to existing learning curves from fixed sets and largely precludes the use of MASIF on learning curves sampled randomly from continuous search spaces.  This is more a limitation of the algorithm selection setting studied than the methodology itself.\n- The performance of MASIF is not significantly better than non-metalearned baselines while requiring significant meta-training data to learn the MASIF model.  While MASIF does well on Synthetic and TaskSet, Scikit-CC18 and LCBench are arguably more reflective of the types of tuning tasks a data scientist would perform in practice. On these, benchmarks, there isn't that large of a gap (~0.02) between MASIF and IMFAS and SH.  In terms of meta-training data, the cost I'm referring to is that of collecting learning curves for tasks in the meta-training set which can be expensive if starting from scratch.  \n- ~~I have many reservations about the experiments.~~  In particular, the baselines should be stronger. ~~SH results look questionable for TaskSet, there are a lot of seemingly arbitrary decisions on the included tasks and hyperparameters for the benchmarks studied.~~\n- ~~The writing is confusing and can be improved.   In particular, I highly recommend the authors change the phrasing \"myopic/myopia\". I also find claims about new multi-fidelity meta-learning setting to not be well supported (see [Multi-task Learning Curve Forecasting Across\nHyperparameter Configurations and Datasets](https://2021.ecmlpkdd.org/wp-content/uploads/2021/07/sub_1011.pdf)).~~\n\n**Post author response**\nThanks for responding to my questions and comments and I recognize I made a few unfair comparisons between the algorithm selection setting and HPO.  I have edited my review accordingly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new meta-learning method for algorithm selection called MASIF that uses multi-fidelity learning curves from previously seen tasks and associated meta-features to learn a model for rankings across a fixed set of hyperparameters.  This model can be used for new tasks over the same hyperparameter space to predict hyperparameter rankings given partial learning curves to speed up algorithm selection.  In contrast to prior work, MASIF does not rely solely on cheap to compute landmark features to inform selection of new configurations nor make parametric assumptions about learning curve for future extrapolation. The authors evaluate MASIF on 4 different benchmarks against established baselines and show MASIF to yield competitive performance across all benchmarks.  Finally, the authors discuss how MASIF can be used in a data scientist's typical workflow to inform budget-regret decisions on which configurations to continue training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Strengthen: add better baselines like [BOHB](https://arxiv.org/abs/1807.01774) and [Learning Curve Prediction with Bayesian Neural Networks](https://openreview.net/pdf?id=S11KBYclx).  I understand that the problem solved by BOHB is different since it can sample from the search space but this is a strictly harder problem than selecting from a finite set as MASIF is doing.  I can imagine running BOHB with different fidelities and giving it the same number of tries as the finite set considered by MASIF and report the final performance of the best found configuration.  \n- ~~Critical: run a version of SH/BOHB that uses the same resources as that given to MASIF.  For example, MASIF sees learning curves trained on a total of # configurations * fidelity, which will be the resources allocated to SH/BOHB.~~\n- ~~Critical: please check result for SH on Taskset.  SH is able to learn something useful on that benchmark according to Figure 3 of [this paper](https://arxiv.org/pdf/2202.09774.pdf).~~\n- Strengthen: please evaluate MASIF on Taskset with all datasets included instead of just the selected tasks with MLP on image datasets.\n- ~~Critical: please provide clarification on TaskSet search space.  The meta-dataset is described as 1000 configurations on 100 tasks in the appendix, does this mean 10 configurations per task or 1000 configurations per task? A search space with 10 configurations in it seems highly unrealistic.~~\n- ~~Critical: please provide clarification on the Scikit-CC18 search space.  Namely, what is the size of the algorithm space A?~~\n- ~~Critical: I am concerned about the practical application of MASIF to real world search spaces and historical experiments.  To address this concern, I think it is critical to run the additional experiments suggested below:~~\n  - ~~Please perform a scaling study of MASIF with larger sets of hyperparameters, e.g. 1000, it seems like experiments only go up to 170 for LCBench.~~\n  - ~~Please perform a study of MASIF with incomplete hyperparameter sets for some tasks, e.g. size of A is 1000 but tasks observe learning curves for say 100 configs.~~", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EPxdsBfrh6", "reviewer": "Reviewer_UzPx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n- The proposed algorithm brings a new perspective to the problem of AS\n- The usage of learning curves for AS is novel but very intuitive\n- The improvement of the proposed algorithm over the existing method is significant.\n\n**Weakness**\n- The paper’s writing quality can be improved. It has many terminologies which are already hard to keep track of and sometimes the notations are inconsistent, making the paper really hard to read. It would benefit from some cleaning up.\n- The proposed architecture is quite complicated and there are not sufficient ablation studies to demonstrate the effects of each component of the architecture. For example, for block 4 of figure 2, why is the algo feature embedding used as a positional encoding rather than the more conventional one-hot encoding? Another example is the usage of guided attention, which is also never ablated. These ablation studies are quite important for algorithms as complicated as this as we can see in the appendix that the utility of block 1 (meta-features) is actually unclear so it would be good to know what each part does.\n- See the Requested changes for specifics.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new algorithm for Algorithm Selection (AS) where the goal is to predict the relative performance of different learning algorithms without the need to execute the whole algorithm fully. This process usually involves using cheap-to-compute features of the data and learning algorithms that contain information about the final performance of the algorithms. The proposed framework MASIF (Meta-learned Algorithm Selection using Implicit Fidelity) additionally leverages the training information from different learning algorithms to meta-learn relationships between different features and the final performance of the model. MASIF leverages a specially designed transformer architecture that takes in information about the data, algorithm hyperparameters and values of the training loss curve and predicts a score that is used to rank different learning algorithms. The architecture takes into a set of learning algorithm features and outputs each algorithm’s score and a ranking loss (between the ranking of predicted scores and true ranking) is reduced via SGD. Experiments show that MASIF outperforms various baselines on 3 benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The definition of $\\mathcal{D}$ is never explicitly specified.\n- Equations 3, and 4 are wrong or at least inconsistent. $s(D)$ outputs an algorithm, but $\\mathcal{L}$ takes in a ranking.\n- Equation 3 also never specifies what the loss for ranking is and I can't see to find this information in the appendix either.\n- Line 84 $m$ is different from line 138 $m$ (i.e., they have different arguments).\n- Elaborate why the algorithm features are treated as position encodings or have an appropriate ablation study\n- Have more ablation studies about different components of the transformer architecture\n", "broader_impact_concerns": "The paper studies the general problem of algorithm selection, so there is no obvious ethical concern as far as I can tell. On the other hand, it could inherit most standard problems in data-driven machine learning such as amplifying bias in the data, etc.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6st0v9IDto", "reviewer": "Reviewer_p2g7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Paper is well-written but verbose. For example, the MASIF architecture description is lengthy and could have gone in the appendix, and presumably the reader does not need to be told things such as “The attention guided transformer-encoder layer returns a tensor of the same dimension as its input. This layer is repeated N times.” Section 3.3 could likely be a single sentence. However, information that I would expect to be in the model description is missing, such as the actual sizing of the layers and weights used (these can be found if you dig into the codebase, but a brief summary of the model size would be very useful in the paper).\n\nOverall I think the paper is fine enough for acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors use a Transformer model they call MASIF to rank machine learning training algorithms by how effective they are predicted to be on a new task. They validate this setup on several datasets of machine learning tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Was “guided attention” necessary to get good performance, or could you have used a standard architecture? Using a more common model could be useful because you can more easily take advantage of optimized implementations.\n\nI find it suspicious that you moved the experiments where your method does not perform as competitively to the appendix (A.3). I don’t think it’s a requirement that a proposed method be SOTA on all experiments run, so I think this should go in the main text.\n\nNothing about how you actually trained the MASIF model was included in the paper, information which should absolutely be present for any paper doing deep learning experiments. Eventually I found https://anonymous.4open.science/r/MASIF-824D/configs/trainer/basetrainer.yaml which shows Adam with an LR=1e-3. Did you tune this at all? I think it’s very fair to try different settings of your algo selector hparams to try to get better performance. Also are you using a learning rate schedule for training MASIF? If not, I _highly_ recommend using a linear warmup for 5% of training followed by a cosine decay, as these seem to be universally useful for Transformer training (at least compared to a constant LR).\n\nLooking at your model configs https://anonymous.4open.science/r/MASIF-824D/configs/model/masif_m_transformer.yaml I see you use a dropout=0.2. Did you tune this? Typically 0.1 is used, and that may give better performance, and unless you’re really worried about overfitting you could even turn this off entirely.\n\nHow did you pick the model sizes? If you are training on all 29 million task set curves (https://github.com/google-research/google-research/tree/master/task_set), then you may be able to get performance gains by increasing the model size beyond the 2-3 128/256/512 dimensional layers I see in the configs https://anonymous.4open.science/r/MASIF-824D/configs/model/masif_transformer.yaml. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to AS that uses transformers meta-learned on task features, algorithm features, and learning curves at different fidelities in order to learn to rank algorithms for a given task. This use of transformers for algorithm selection is quite original, and removing the need for parametric assumptions on the learning curve is very interesting. The reviewers had a number of concerns around verbosity, ablations, and additional benchmarks that were largely resolved through the discussion process.

There was some reservation during the final discussion about the usefulness of the approach in terms of the AS setup. The argument is that the use of a fixed set of hyperparameters is not very representative of the workflow of a typical data scientist. A comparison or extension to multi-fidelity HPO is certainly worth thinking about, however it was felt that the paper is still worth publishing with this setup as it may pave the way for future work in this area.
Decision: Accept (poster)