DeepPipe: Deep, Modular and Extendable Representations of Machine Learning Pipelines
Authors: Sebastian Pineda Arango
Josif Grabocka
Venue: Submitted to ICLR 2023 (2023)
Keywords: Pipeline optimization
meta-learning
bayesian optimization
representation learning

Abstract: Finding accurate Machine Learning pipelines is essential in achieving state-of-the-art AI predictive performance. Unfortunately, most existing Pipeline Optimization techniques rely on flavors of Bayesian Optimization that do not explore the deep interaction between pipeline stages/components (e.g. between hyperparameters of the deployed preprocessing algorithm and the hyperparameters of a classifier). In this paper, we are the first to capture the deep interaction between components of a Machine Learning pipeline. We propose embedding pipelines in a deep latent representation through a novel per-component encoder mechanism. Such pipeline embeddings are used with deep kernel Gaussian Process surrogates inside a Bayesian Optimization setup. Through extensive experiments on large-scale meta-datasets, we demonstrate that learning pipeline embeddings with Deep Neural Networks significantly advances the state-of-the-art in Pipeline Optimization.

Under review as a conference paper at ICLR 2023
DeepPipe: DEEP, MODULAR AND EXTENDABLE REPRESENTATIONS OF MACHINE LEARNING PIPELINES
Anonymous authors Paper under double-bling review
ABSTRACT
Finding accurate Machine Learning pipelines is essential in achieving state-of-the art Al predictive performance. Unfortunately, most existing Pipeline Optimization techniques rely on flavors of Bayesian Optimization that do not explore the deep interaction between pipeline stages/components (e.g. between hyperparameters of the deployed preprocessing algorithm and the hyperparameters of classifier). In this paper, we are the first to capture the deep interaction between components of Machine Learning pipeline. We propose embedding pipelines in deep latent representation through novel per-component encoder mechanism. Such pipeline embeddings are used with deep kernel Gaussian Process surrogates inside a Bayesian Optimization setup. Through extensive experiments on three large- scale meta-datasets, including Deep Learning pipelines for computer vision, we demonstrate that learning pipeline embeddings achieves state-of-the-arr results in Pipeline Optimization.
INTRODUCTION
Machine Learning (ML) has proven to be successful in wide range of tasks such image clas- sification, natural language processing, and time series forecasting. In a supervised learning setup, practitioners need define a sequence of stages comprising algorithms that transform the data (e.g. imputation, scaling) and produce an estimation (e.g. through classifier or regressor). The selection of the algorithms and their hyperparameters. known as Pipeline Optimization (Olson & Moore 2016) or pipeline synthesis (Liu et al. 2020 Drori et al. 2021) is challenging. Firstly, the search space contains conditional hyperparameters, as only some of them are active depending on the selected algorithms. Secondly, this space is arguably bigger than the one for single algorithm. Therefore, previous work demonstrates how this pipeline search can be automatized and achieve competitive results (Feurer et al.) 2015 Olson & Moore 2016). Some of these approaches include Evolutionary Algorithms (Olson & Moore, 2016), Reinforcement Learning (Rakotoarison et al. 2019 Drori et al.) 2021 Bayesian Optimization (Feurer al. 2015 Thornton al. [012 Alaa & van der Schaar
2018
Pipeline Optimization (PO) techniques need capture the complex interaction between the algorithms of a Machine Learning pipeline and their hyperparameter configurations. Unfortunately. no prior method uses Deep Learning to encapsulate the interaction between pipeline components. Prior work trains performance predictors (a.k.a. surrogates) on the concatenated hyperparameter space of all algorithms (raw search space), for instance, using random forests (Feurer al., 2013) or finding groups of hyperparameters to use on kernels with additive structure (Alaa & van der Schaar 2018). On the other hand, transfer learning has been shown decisively improve PO by transferring efficient pipelines evaluated on other datasets (Fusi et al.) 2018 Yang et al. 2019 2020). Our method is the first to introduce a deep pipeline representation that meta-learned to achieve state-of-the-ar results in terms of the quality of the discovered pipelines.
We introduce DeepPipe, neural network architecture for embedding pipeline configurations on a latent space. Such deep representations are combined with Gaussian Processes (GP) for tuning pipelines with Bayesian Optimization (BO). We exploit the knowledge of the hierarchical search space of pipelines by mapping the hyperparameters of every algorithm through algoritht encoders to a hidden representation, followed by a Fully Connected Network that receives the concatenated representations as input, Additionally, we show that meta-learnigg this network through evaluations

---

Under review as a conference paper at ICLR 2023
on auxiliary tasks improves the quality of BO. Experiments three large-scale meta-datasets show that our method achieves the new state-of-ther tof-the-art in Pipeline Optimization.
Our contributions are as follows:
We introduce DeepPipe, a surrogate for BO that achieves peak performance when optimizing pipeline for new dataset through transfer learning.
We present a novel and modular architecture that applies different encoders per stage and yields better generalization in low meta-data regimes, i.c. few/no auxiliary tasks.
We conduct extensive evaluations against seven baselines on three large meta-datasets, and we further compare against rival methods in OpenML datasets to assess their performances under time constraints.
We demonstrate that our pipeline representation helps achieving state-of-the-art results in optimizing pipelines for fine-tuning deep computer vision networks.
2 RELATED WORK
Full Model Selection (FMS) is also referred to as Combined Algorithm Selection and Hyperparame- ter optimization (CASH) (Hutter et al. 2019 Feurer et al. 2015). FMS aims to find the best model and its respective hyperparameter configuration (Hutter et al. 2019). A common approach is to use Bayesian Optimization with surrogates that can handle conditional hyperparameters, such as Random Forest (Feurer et al. 2013), tree-structured Parzen estimators (Thornton et al. 2012). or ensembles of neural networks (Schilling et al. 2015).
Pipeline Optimization (PO) is generalization of FMS where the goal is to find the algorithms and their hyperparameters for different stages of Machine Learning Pipeline. Common approaches tree structure and use reinforcement learning (Rakotoarison al. 2019
model the search space
Drori et al. 2021 de Sá et al. (017), evolutionary algorithms (Olson & Moore (016)). Hierarchical Task Networks (Mohr et al. (2018) for searching pipelines. Other approaches
Multi-Armed Bandit
strategies to optimize the pipeline, and combine them with Bayesian Optimization (Swearingen
al. 2017) or multi-fidelity optimization (Kishimoto et al. (02T). Alaa & van der Schaar (2018) use additive kernels on Gaussian Process surrogate to search pipelines with BO that groups the algorithms in clusters and fit their hyperparameters on independent Gaussian Processes, achieving an effectively lower dimensionality per input. By formulating the Pipeline Optimization constrained optimization problem, Liu (Liu et al.) 2020) introduce a method based on alternating direction method of multipliers (ADMM) (Gabay & Mercier 1976).
Transfer-learingg for Pipeline Optimization and CASH leverages information from previous (auxiliary) task evaluations. Few approaches use dataset meta-features to warm-start BO with good configurations from other datasets (Feurer et al. 2015) Alaa & van der Schaar 2018). As extracting meta-features demands computational time, follow-up up works find portfolio based these auxiliary tasks (Feurer et al. (020). Another popular approach is to use collaborative filtering with a matrix of pipelines vs task evaluations learn latent embeddings of pipelines. OBOE obtains the embeddings by applying a QR decomposition of the matrix on a time-constrained formulation (Yang et al.) 2019)). By recasting the matrix tensor, Tensor- OBOE (Yagg et al. (020)) finds latent representations via the Tucker decomposition. Furthermore, Fusi et al. 2018) apply probabilistic matrix factorization for finding the latent pipeline representations. Subsequently. they use the latent representations as inputs for Gaussian Process, and explore the search space using BO.
3 PRELIMINARIES
3.1 PIPELINE OPTIMIZATION
The pipeline of a ML system consists of sequence of N stages (e.g. dimensionality reducer, standardizer, encoder, estimator (Yang et al. 2020)). At each stage i E N) a pipeline includes one algorithm from a set of M, choices (e.g. the estimator stage can include the algorithms
AAttMM. systems might select multiple algorithms in stage, however, solution trivially generalizes by decomposing stages into new sub-stages with only subset of algorithms.
2

---

Under review as a conference paper at ICLR 2023
(SVM, MLP, RF1). Algorithms are tuned through their hyperparameter search spaces, where As.j denotes the configuration of the j-th algorithm in the i-th stage. Furthermore, let us denote a pipeline p the set of indices for the selected algorithm at each stage, i... p = (PI....., ,N), where Pi E {1... M,} represents the index the selected algorithm at the i-th pipeline stage. The hyperparameter configuration of pipeline is the unified set of the configurations of all the algorithms in pipeline, concretely X(p) == (A1.p1..... AN ). Ai.p E p.... Pipeline Optimization demands finding the optimal pipeline p' and its optimal configuration X(p*) by minimizing the validation loss
of trained pipeline on a dataset 'D as shown in Equation
"0',((")))
arg min PE(1...M; Mi xx-x M App)AM1 x- X-NN
L (p. A(p), D)
From now we will use the term pipeline configuration for the combination of a sequence of algorithms p and their hyperparameter configurations X(p), and denote it simply px == (p)).
3.2 BAYESIAN OPTIMIZATION
Bayesian optimization (BO) is a mainstream strategy for optimizing ML pipelines (Feurer et al. 2015)
Hutter et al. 2011 Alaa & van der Schaar 2018 Fusi et al. [001 Schilling et al., 2015). Let us start
with defining history of Q evaluated pipeline configurations as H. = {(x y)),,, ( ( ))
where (/9) ~ ((((p) (()) og)) is a probabilistic modeling of the validation loss ((x)0) achieved with (a)
the q-th evaluated pipeline configuration P) , {1... Q}. Such a validation loss is approximated with a surrogate model, which is typically a Gaussian process (GP) regressor. We measure the similarity between pipelines via kernel function k dom (px) dom (px) RR parameterized
with 0, and represent similarities as a matrix K2 == k(p))) (9) (0)
.PX ;7), K' E RQx Since we consider noise, we define K K' + o,I GP estimates the validation loss f. of a new pipeline configuration px (*) by computing the posterior mean E [fx]
and posterior variance V [f*] as:
E fa Px (+) n1 KK [ 1xx"), H. -K.. - -K-
where K., = (px px) :7) K. E Ro. and K.. = & .x K., E R
BO is an iterative process that alternates between fitting GP regressor described above and selecting the next pipeline configuration to evaluate (Snoek et al. 2012). A description of how BO finds pipelines using GP surrogate is provided in Appendix !
DeepPipe: BO WITH DEEP PIPELINE CONFIGURATIONS
To apply BO to Pipeline Optimization (PO) we must define a kernel function that computes the similarity of pipeline configurations, i.e. &o
?. Prior work exploring BO for PO use kernel functions directly on the raw concatenated vector space of selected algorithms and their hyperparameters (Alaa & van der Schaar 2018 or use surrogates without dedicated kernels for the
px) PA (8) (0:00
(g)
conditional search space (Feurer et al. 2015:Olson Olson & Moore 2016 Schilling et al. 2015).
However, we hypothesize that these approaches cannot capture the deep interaction between pipeline stages, between algorithms inside stage, between algorithms across stages, and between different configurations of these algorithms. In order address this issue we propose simple, yet powerful solution to PO: learn deep embedding of pipeline configuration and apply BO with deep kernel (Wistuba& & Grabocka 2021; Wilson al.. 2016).
This is done by DeepPipe, which searches pipelines in latent space using BO with Gaussian Processes. We use a neural network d(px; dom(px) - TRZ with weights 0 to project pipeline configuration to a Z-dimensional space. Then, we measure the pipelines" similarity in this latent
space as k: ( (0; 00 (0) (2;0))
:)) using the popular Matérn 5/2 kernel (Genton 2002). Once we
3

---

Under review as a conference paper at ICLR 2023
compute the parameters of the kemel similarity function, compute the GP's posterior and conduct PO with BO as specified in Section 3.2
In this work, we exploit existing deep kernel learning machinery (Wistuba & Grabocka 202 Wilson ctt al. 2016) to train the parameters 9 of the pipeline embedding neural network Φ, and the parameters y of the kernel function k, by maximizing the log-likelihood of the observed validation losses y of the evaluated pipeline configurations Px- The objective function for training deep kernel the log marginal likelihood of the Gaussian Process (Rasmussen & Williams/ 2006) with covariance matrix entries kige = k ( ((p))(px))) :0).
4.1 PIPELINE EMBEDDING NETWORK
Preprocessing Stage
Min.
The main piece of the puzzle is: How to define the pipeline configuration embedding o²
(0.))
Stago
With Mean
(
Our DeepPipe embedding is com- posed of two parts (i) per-algorithm encoders, and (ii) a pipeline aggrega- tion network. A visualization exam- ple our DeepPipe embedding archi- tecture is provided in Figure [I We define one encoder ((b)) for the hy- perparameter configurations of each j-th algorithm, in each i-th stage, as plain multi-layer perceptrons (MLP). These encoders, each parameterized by weights (ene map the algorithms' configurations to L--dimensional vector space:
Chasification Stage
g(a.))
Conventions
Kemee
Stage
Trees
Hyperparameters
(
MLP
Depth
Selector
Weight Decay
Concatenator
Learning Rase
Figure 1: An example architecture for DeepPipe search space with 2 stages (Preprocessing, Classification).
(155) (Ar,; = MLP (Ar.s 0:g)) c(t)) Arj + +R E NJ, E M. (3)
For a pipeline configuration px, represented with the indices its algorithms and the configuration vectors of its algorithms A(p), we project all the pipeline's algorithms' configurations their latent space using the algorithm-specific encoders. Then, we concatenate their latent encoder vectors, where concatenation notation is R4, @ LR I +L, Finally, the concatenated representation is
embedded to final I& space via an aggregating MLP ab with parameters (fee as denoted below:
@ (pa)= ((()))))))) e........ .........ee...... (NNN (AN.px) (AREE)) & W: -R
:
(4)
Within the i-th stage, only the output of one encoder concatenated, therefore the output of the Selector corresponds to the active algorithm in the i-th stage and can be formalized as ((i. Σje [[j = a) g(t.)) (As,)), where a is the index the active algorithm and denotes the indicator function. Having defined the embedding & in Equations 3 we can plug it into the kernel function, optimize minimizing the negative log likelihood of the GP with respect
{@ene ( and
conduct BO as in Section
4.2 META-LEARNING OUR PIPELINE EMBEDDING
In many practical applications, there exist computed evaluations of pipeline configurations on previous datasets, leading the possibility of transfer learning for PO. Our DeepPipe can be easily meta-learned from such past evaluations by pre-training the pipeline embedding network. Let denote the meta-dataset of pipeline evaluations on T datasets (a.k.a. auxiliary tasks) as Hi ((px (f.1), ((..1)). (px (t.Q.), (..0:), (((..))), & (1,...; T), where Q. is the number of existing
4

---

Under review as a conference paper at ICLR 2023
evaluations for the t-th dataset. As result, meta-learn our method's parameters minimize the meta-learning objective of Equation[5] This objective function corresponds to the negative marginal likelihood of the Gaussian Processes using DeepPipe's extracted features as imput to the kernel (Wistuba & Grabocka 2021] Patacchiola et al. 2020). Further details on the meta-learning procedure of our pipeline configuration weights are provided in Appendix
y (10)0,))~')))) log (0,0)
arg min
(5)
7.8
EXPERIMENTS AND RESULTS
5.1 META-DATASETS
preta-dtaset a collection of pipeline configurations and their respective performance evaluated in different tasks (i.e. datasets). Information about the meta-data sets is provided on Appendix PP their search spaces are clarified on Appendix R and the splits of tasks per meta-dataset are found on Appendix[] T All the tasks in the meta-dataset correspond to classification. We use the meta-training set for pre-training the Pipeline Optimization (PO) methods, the vela-vaaio set for tuning some of the Syper--arameters of the PO method, and we assess their performance the meta-test set. In our experiments, we use the following meta-datasets.
PMF contains 38151 pipelines (after filtering out all pipelines with only NaN entries), and 553 datasets (Fusi et al., 2018). Although not all the pipelines were evaluated all tasks, it has a total of 16M evaluations. The pipeline search space has stages (preprocessing and estimator) with 2 and 11 algorithms respectively. Following the setup in the original paper (Fusi et al. 2018). we take 464 tasks for meta-training and 89 for meta-test. As the authors do not specify validation meta-dataset, we sample randomly 15 tasks out of the meta-training dataset.
TenSor-OBOE provides 23424 pipelines and 551 tasks (Yang al., 2020 It contains 1 11M evalua- tions, as there exist no evaluations for some combinations of pipelines and tasks. The pipelines include 5 stages:
Reeuucr Imputator (1 algorithm),, Dimensionality Reducer (3 algorithms), Standardizer (1 algorithm).
Encoder (I algorithm), and Estimator (11 algorithms). We assign 331 tasks for meta-trainin tasks, 110 tasks for meta- validatoi,,,,nn, and 110 tasks for meta-testing.
ZAP is benchmark that evaluates deep learning pipelines on fine-tuning state-of-the-arr computer vision tasks (Ozturk al. 2022). The meta-dataset contains 275625 evaluated pipeline configurations on 525 datasets and 525 different Deep Learning pipelines (i.e. the best pipeline of dataset was evaluated also on all other datasets). From the set of datasets, we use 315 for meta-training, 45 for meta-validation and 105 for meta-test, following the protocol of the original paper.
In addition, we use OpenML datasets. It comprises curated datasets (Gijsbers al., 2019) and has been used in previous work for benchmarking (Erickson et al., (020). This dataset does not contain pipeline evaluations like the other three meta-datasets above. However, we use the OpenML collection for evaluating the Pipeline Optimization in time-constrained settings (Ozturk et al. 2022)
5.2 BASELINES
Random Search (RS) selects pipeline configurations by sampling randomly from the search space (Bergstra & Bengio 2012).
Probabilistic Matrix Factorization (PMF) uses a surrogate model that learns latent representation for every pipeline using meta- veta-training tasks (Fusi et al. 2018). We tuned this latent dimension for the Tensor-OBOE dataset from grid of {10, 15, 20) and found 20 to be the best setting. For the PMF-Dataset, where the model was introduced, we used the default value of 20. We use the original PMF implementation (Sheth, 2018).
OBOE also uses matrix factorization for optimizing pipelines, but they aim to find fast and informative algorithms to initialize the matrix (Yang et al. 2019). We use the settings provided by the authors.
5

---

Under review as a conference paper at ICLR 2023
Tensor-OBOE formulates PO tensor factorization, where the rank of the tensor is equal to 1 + N, for N being the number of stages in the pipeline (Yang et al., 2020). We tuned the rank for the Tucker decomposition from grid of (20.30,40), resulting in the best value being 30. All the other hyper-parameters were set as in the original implementation (Yng et al. 2019).
Factorized Multilayer Perceptron (FMLP) creates ensemble of neural networks with factorized layer (Schilling et a 2013). The inputs of the neural network are the one-hot encodings of the algorithms and datasets, in addition the algorithms' hyperparameters. We tuned the number of base estimators for the ensemble from a grid {10, 50, 100), with 100 being the optimal ensemble size.
Each network layer has 5 neurons and ReLU activations highlighted in the author's paper (Schilling et al. 2015).
RGPE builds an ensemble of Gaussian Processes using auxiliary tasks (Feurer al. 2018). The ensemble weights the contributions of every base model and the new model fit on the new task. We used the implementation from Botorch (Balandat al. 2020).
Gaussian Processes (GP) are standard and strong baseline in hyperparameter optimization (Snoek ct al. 2012). We tuned the kernel from (Gaussian, Matérn 5/2), with Matérn 5/2 performing better.
DNGO uses neural networks as basis functions with a Bayesian linear regressor at the output layer (Snoek et al. 2015). We use the implementation provided by Klein & Zela (2020), and its default hyperparameters.
SMAC uses Random Forest for predicting uncertainties (Hutter et al. 2011). After exploring a grid of (10, 50, 100} for the number of trees, we found 100 to be the best choice.
TPOT an AutoMI. system that conducts PO using evolutionary search (Olson & Moore 2016). We use the original implementation but adopted the search space to fit the Tensor-OBOF meta-dataset
(see Appendix R
5.3 RESEARCH HYPOTHESES AND ASSOCIATED EXPERIMENTS
Hypothesis 1: DeepPipe outperforms standard PO basclines.
Experiment 1:
We evaluate the performance of DeepPipe when no meta- data is available.
We compare against four baselines: Random Search (RS) (Bergstra et al.) 2011). Gaussian Processes
(GP) (Rasmussen & Williams) 2006), DNGO (Snoek et al. 2015) and SMAC (Hutter et al., 2011).
We evaluate their performances on the aforementioned PMF, Tensor-OBOE and ZAP meta-datasets.
In Experiments and 2 (below) we select 5 initial observations to warm-start the BO, then we run 95 additional iterations. The procedure for choosing these configurations is detailed in the Appendix G
Hypothesis 2: Our meta-learned DeepPipe outperforms state-of-the-ar transfer-leamiing PO methods.
Experiment 2: We compare our proposed method against baselines that use auxiliary tasks (a.k.a. meta-training data) for improving the performance of Pipeline Optimization: Probabilistic Matrix Factorization (PMF) (Fusi al. 2018), Factorized Multilayer Perceptron (FMLP) (Schilling et al.)
2015). OBOE Yang et al. 2019) and Tensor OBOE (Yang et al.) (020)). Moreover, we compare to RGPE (Feurer al.) 2018), an effective baseline for transfer HPO (Arango et al. 202T) We evaluate the performances on the PMF and Tensor-OBOF meta-datased
Hypothesis 3: DeepPipe leads to strong any-time results in a time-constrained PO problem.
Experiment 3: Oftentimes practitioners need AutoML systems that discover efficient pipelines within
small time budget. To test the convergence speed of our PO method we ran on the aforementioned OpenML datasets for a budget of 10 minutes, and also hour. We compare against five baselines: (i) TPOT (Olson& & Moore 2016) adapted to the search space of Tensor-OBOE. (see Appendix R). (ii) OBOE and Tensor-OBOl Yang et al. [009 2020) using the time-constrainee version provided by the authors, (iii) SMAC (Hutter et al.) 2011). and (iv) PMF (Fusi et al. 2018). The last three had the same five initial configurations used to warm-start BO detailed in Experiment 1. Moreover, they were pre-trained with the Tensor-OBOE meta-dataset and all the method -pecificc settings are the same as in Experiment 2.
Hypothesis 4: Our novel encoder layers of DeepPipe enable an efficient PO when the pipeline search space changes, i.e. when developers add new algorithm to ML system.

---

Under review as a conference paper at ICLR 2023
Tensor OBOE Meta Dataset
PMF Meta Datases
ZAP Meta Datasel
4.0
Average Rank
3.5
3.5
3.2
3.0
3.0-
3.0
2.8
2.5
20
10
S0
100
20
40
00
80
100
20
40
60
80
100
No. of Expllored Pipelines
No. of Explored Pipelines
No. of Explored Pipelines
DeepPipe
GP
DNGO
SMAC
Figure Comparison of DeepPipe vs. standard PO methods (Experiment
TensorOBOO Meto-Datused
PMF Metu-Dutuset
ZAP Metu-Datuse:
4.5
Average Bank
4.0-
3.5-
3.5-
3.0-
3.04
2.5
20
40
60
80
100
5
20
40
60
80
100
5
20
40
00
80
100
No. of Explored Pipelines
No. of Explored Pipeliness
No. of Explored Pipolines
DeepPipe
PMF
FMLP
T-OBOE
RS
OBOE
RGPE
Figure 3: Comparison of DeepPipe vs. transfer-learning PO methods (Experiment 2)
Experiment 4: major obstacle meta-learning PO solutions that they do generalize when the search space changes, especially when the developers of ML systems add new algorithms. Our architecture quickly adapts to newly added algorithms because only an encoder sub-network for the new algorithm should be trained. To test the scenario, we ablate the performance of five versions of DeepPipe and try different settings when we remove a specific algorithm (an estimator) either from meta-traine trai-training, meta-testing, both.
Hypothesis The encoders in DeepPipe introduce an inductive bias where latent representa- tion vectors of an algorithm's 's configurations are co-located, and located distantly from the repre- sentations of other algorithms' configurations. Formally, given three pipelines (9), (mm) p( if
P + (T then [lp()")) Φ(p (m)( [lø(p(m)) ((p())) with higher probability when using encoder layers, given that p; (n) is the index of the algorithm in the i-th stage. Furthermore, we hypothesize that the less number tasks during pre-training, the more necessary this inductive bias is.
p P;
(m)) (I)
Experiment 5: We sample 2000 pipelines of 5 estimation algorithms on the TensorOBOE dataset. Subsequently, embed the pipelines using DeepPipe with 0, 1 and 2 encoder layers. and weights 8, initialized such that 0; 0 are independently identically distributed 0; ~ AN 1). Finally, visualize the embeddings with T-SNE (Van der Maaten & Hinton 2008) and compute a cluster metric to assess how close pipelines with the same algorithm are in the latent space: E,00 pit)) pln; (I()))(")) - (( < [[((p(m)) - ((p()))l)) To test the importance of the inductive bias the number of pre- pre-training tasks, ablate the performance of DeepPipe for different percentage of pre- training tasks (0.5%, 1%, 5%, 10%, 50%, 100%) under different values of encoder layers.
7

---

Under review as a conference paper at ICLR 2023
5.4 EXPERIMENTAL SETUP FOR DeepPipe
The encoders and the aggregated layers are Multilayer Perceptrons with ReLU activations. We keep an architecture that is proportional to the input size. The number of neurons in the hidden layers for the encoder of algorithm j-th in ě-th stage with |As,) hyperparameters is F P- given an integer factor F. The output dimension of the encoders of the i-th stage is defined as Qi = max,| |Ai,fl The number of total layers (i.e. encoder and aggregation layers) is fixed to 4. The number of encoders is chosen from [0.1,2). The specific values of the encoders' input dimensions are detailed in Appendix R We choose F E {4, 6, 8, 10} based on the performance in the validation split. Specifically, we use the following values for DeepPipe: (i) in Experiment
encoder layer (all meta-datasets), F
(PMF and ZAP) and F = 8 (Tensor-OBOE), (it) in Experiment 2: F = 8, no encoder layer (PMF, F
Tensor-OBOE) and one encoder layer (ZAP), (ii) in Experiment
and no encoder layers, (iv) in Experiment 4 we use F = 8 and {0, }} encoder layers. Finally (iv) in Experiment 5 we use F = 8 and {0, 1, 2} encoder layers. Additional details on the setup can be found in the Appendix G and our source code
RESULTS
We present the results for Experiments and Figures 2 and 3 respectively. both cases, we compute the ranks of the classification accuracy achieved by the discovered pipelines of each technique, averaged across the meta-testing datasets. The shadowed lines correspond the 95% confidence intervals. Additional results showing the mean regret are included in Appendix L In Experiment (standard/non-transtte PO) DeepPipe achieved the best performance for both meta- datasets, whereas SMAC attained the second place. In Experiment 2 DeepPipe strongly outperforms all the transfer-learning PO baselines in all meta-datasess. Given that DeepPipe yields state-of-the-ap PO results on both standard and transfer-learinng setups, we conclude that our pipeline embedding network computes efficient representations for PO with Bayesian Optimization. In particular, the results the ZAP meta-dataset indicate the efficiency of DeepPipe in discovering state-of-the-ar Deep Learning pipelines for computer vision.
Experiment 3 conducted on the Table I: Average Rank of Accuracy on OpenML Datasets OpenML datasets shows that Deep- Pipe performs well under restricted budgets (see Ta.le Although our method does not incorporate any di- rect way to handle time constraints, is interesting to note that it out- performs other methods that include heuristics for handling a quick con- vergence, such as OBOE and Tensor- OBOE.
Method
10 Mins
Hour
TPOT
3.2000 + 0.1998
3.3527 + 0.191T
Tensor- OBOE
4.3878 = 0.1786
4.3624 = 0.2008
OBOE
3,9909 = 0.1935
4.0873 0.2072
SMAC
= 0.1663
3.1682 0.1484
PMF
3.0424 = 0.1577
2.9336 = 0..123
DeepPipe
2.7424 + 0.1277
2.8996 0.1343
Furthermore, the results of Experiment 4 indicate that our DeepPipe embedding quickly adapts to incrementally-exxaading search spaces, e.g. when the developers of a ML system add new algorithms. In this circumstance, existing transfer-learning PO baselines do not adapt easily, because they assume a static pipeline search space. As a remedy, when new algorithm is added to the system after meta-learning our pipeline embedding network, we train only a new encoder for that new algorithm. In this experiment, we run our method on variants of the search space when one algorithm at a time introduced to the search space (for instance an estimator, e.g. MLP, RF, etc., not known during meta-training, but added new to the meta-testing).
In Tables 2 and 5 (in Appendix), we present the values of the average rank among five different configurations for DeepPipe. We compare among meta-trained versions (denoted by / the column MTd.) that omit specific estimators during meta-training (MTr:=/), or during meta-testing (MTe.=/). We also account for versions with one encoder layer denoted by in the column Enc.
TThe code is available in this repository: DeepPiped 3DDE
https::/ 4open. sci www.ecieccc... ence/r/
8

---

Under review as a conference paper at ICLR 2023
No Encoder Layers
One Encoder Layer
Two Encoder Layers
75
50
25
0
-25
-2
-25
-50
-00
-50
50
-50
Cluster Metric: 0,7252
Cluster Metric: 0.8338
Cluster Metric: 0.9275
ExtraTrees
GBT
Logit
MLP
RF
Figure 4: Embeddings of Pipelines produced by random initialized DeepPipe (after applying T- SNE). The color indicates the active algorithm in the Estimation stage of Tensor-OBOE Meta-Dataset.
The best in all cases is the meta-learned model that did not omit the estimator (i.e. algorithm known and prior evaluations with that algorithm exist). Among the versions that omitted the estimator in the meta-training set (i.e. algorithm added new), the best configuration was the DeepPipe which fine-tuned a new encoder for that algorithm (line Enc=/, E MTd.=V, MTT:=/, =Te..=X). This version of DeepPipe performs better than ablations with encoder layers (i.e. only aggregation layers Φ), or the one omitting the algorithm during meta-testing (i.e. pipelines that do not use the new algorithm at all). The message of the results is simple: If we add instead of running PO without meta-learning (because the search space changes and existing transfer use meta-learned DeepPipe and only PO baselines not applicable to the space), fine-tune an encoder for a new algorithm.
algorithm to ML system,
Table 2: Average rank among DeepPipe variants for newly- algorithms (Tensor-OBol
Omitted in
Omirted Estimator
Enc.
MTd.
MTr.
MTe.
ET
GBT
Logit
MI.P
RF
ISVM
KNN
AB
GB/PE
3.1572
3.4135
3.2646
3.1580
x
V
X
3.5319
3.0934
3.6362
3,4780
3.4712
3.3829
3.6312
3.3691
3.6333
3.4642
X
2.5882
2.6773
2 2088
2.5761
2.6485
2.6938
2.6812
2,5596
2.5936
2.5546
2.9247
3.0743
2.8802
3.0423
2.6691
2.8026
2.7408
2.9161
2.9214
2. 26888
X 2.7455
2.9978
2.7248
2.7054
2.7978
2.7619
2.6822
2.8688
2.6938
2.8007
The effect of the inductive bias introduced by the encoders (Experiment 5) be appreciated in Figure 4 The pipelines with the same active algorithm in the estimation stage, but with different hyperparameters, lie closer in the embedding space created by random initialized DeepPipe, forming compact clusters characterized by the defined cluster metric (value below the plots). We formally demonstrate in Appendix [S] that, in general, single encoder layer creating more compact clusters than fully connected linear layer. In additional results (Appendix [T) we observe that the average rank on the test-tasks improves for DeepPipe versions with deeper encoder layers (keeping the total number of layers fixed), if the number of meta-training tasks gets lower. This occurs because the objective function (Equation (5) makes possible to learn embeddings where pipelines with similar performance clustered together (see Appendix O given enough meta-training data. Otherwise, the inductive bias introduced by the encoders becomes more relevant.
7 CONCLUSION
In this paper, we have shown that efficient Machine Learning pipeline representations can be com- puted with deep modular networks. Such representations help discovering more accurate pipelines compared to the state-of-art approaches, because they capture the interactions of the different pipelines algorithms and their hyperparameters. Moreover, we show that introducing aer-algorithm encoders helps in the case of limited oettner data, when algorithm is added to the search space. Overall, we demonstrate that our method DeepPipe achieves the new state-of-the-ars in Pipeline Optimization.

---

Under review as a conference paper at ICLR 2023
Limitations. Our representation network does not model complex parallel pipelines (in that case the embedding will be a graph neural network), and/or pipelines involving ensembles. We plan to investigate these important points in our future work.
Reproducibility Statement. To guarantee the reproducibility of or work, we include an anonymized repository to the related code. The code for the baselines is included also, and reference the original implementations if it is the case. All the meta-datasets are publicly available and correspondingly referenced.
REFERENCES
Yuji Akimoto and Chengrun Yang. Oboe. hattos//ggttaab..cc////elllooppp
2020.
Ahmed M. Alaa and Mihaela van der Schaar. Autoprognosis: Automated clinical prognostic modeling via bayesian optimization with structured kernel learning. In Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmässan. Stockholm, Sweden, July 10-15, 2018, pp. 139-148, 2018.
Sebastian Pineda Arango, Hadi S. Jomaa, Martin Wistuba, and Josif Grabocka. Hpo-b: A large-scale reproducible benchmark for black-box hpo based on openml, 2021.
Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian optimization. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Mach. Learn. Res., 13:281-305, 2012.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-paramettt optimization. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December
2011, Granada, Spain, pp. 2546-2554, 2011.
Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Mare Peter Deisenroth. Manifold gaussian processes for regression. In 2016 International Joint Conference on Neural Networks (IJCNN), pp. 3338-3345. IEEE, 2016.
Alex Guimarães Cardoso de Sá, Walter José G. S. Pinto, Luiz Otávio Vilas Boas Oliveira, and Gisele L. Pappa. RECIPE: A grammar-based framework for automatically evolving classification pipelines. In James McDermott, Mauro Castelli, Lukás Sckanina, Evert Haasdijk, and Pablo García- Sánchez (eds.), Genetic Programming 20th European Conference, EuroGP 2017, Amsterdam, The Netherlands, April 19-21, 2017, Proceedings, volume 10196 of Lecture Notes in Computer 16. URL htt.httpppp.. /ww.///ooioo// orrg
Science, pp. 261-261, 2017. doi: 3111:/9988--319-5568-- - 100100077776--3111--------------
Iddo Drori, Yamuna Krishnamurthy, Rémi Rampin, Raoni de Paula Lourenço, Jorge Piazentin Ono, Kyunghyun Cho, Cláudio T. Silva, and Juliana Freire. Alphad3m: Machine learning pipeline synthesis. CoRR, abs/21 11.02008. 2021. /ttpssttt////xxit.
abs 2111. 02508
Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon tabuula:: Robust and accurate automl for structured data. arxiv 2020. arXiv preprint arXiv:2003.066: 06505, 2020.
Matthias Feurer, Aaron Klein, Jost Eggensperger, Katharina Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems 28 (2015), pp. 2962-2970, 2015.
Matthias Feurer, Benjamin Letham, and Eytan Bakshy. Scalable meta-learning for bayesian opti- mization using sunking-weighte gaussian process ensembles. In AutoML Workshop at ICML, volume 7, 2018.
10

---

Under review as a conference paper at ICLR 2023
Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto- skleam 2.0: Hands-free free automl via meta-learning. arXiv preprint arXiv:2007.04074 2020.
Nicoló Fusi, Rishit Sheth, and Melih Elibol. Probabilistic matrix factorization for automated machine learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 3352-3361, 2018.
Daniel Gabay and Bertrand Mercier. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers mathematics with applications, 2(1): 17-40, 1976.
Jacob Gardner, Chuan Guo, Kilian Weinberger, Roman Garnett, and Roger Grosse. Discovering and Exploiting Additive Structure for Bayesian Optimization. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence and Statisties, volume 54 of Proceedings of Machine Learning Research, pp. 1311-1310 PMLR. 20-22 Apr 2017. URL
.ml..ssmml... www.ppressss..... sre.oeeeinngg.m...rrsssss///// 2axees//55//aacceeiil html
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXfv:1807.01622. 2018.
Mare G. Genton. Classes of kernels for machine learning: statisties perspective. 2:299-312, mar 2002. ISSN 1532-4435.
Pieter Gijsbers, Erin LeDell, Janek Thomas, Sébastien Poirier, Bernd Bischl, and Joaquin Vanschoren. open source automl benchmark. arXiv preprint arXiv:/907.00000 000900 2019.
Frank Hutter, Holger H. Hoos, and Kevin Seyton-Brown Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers, pp. 507-523, 2011. doi: 40.
0000077788-362--25666-5
Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.). Automated Machine Learning Methods, Systems, Challenges. The Springer Series on Challenges in Machine Learning, Springer, 2019. ISBN 978-3-030-05317-8. doi: 10.
Fergus Imrie, Bogdan Cebere, Eoin McKinney, and Mihaela van der Schaar. Autoprognosis 2.0: Democratizing diagnostic and prognostic modeling in healthcare with automated machine learning. arXiv preprint arXiv:2210. 12090, 2022.
Akihiro Kishimoto, Djallel Bouneffouf, Radu Marinescu, Parikshit Ram, Ambrish Rawat, Martin Wistuba, Paulito Pedregosa Palmes, and Adi Botea. Bandit limited discrepancy search and application to machine learning pipeline optimization. In 8th ICML. Workshop on Automated Machine Learning (AutoML), 2021.
Aaron Klein and Arber Zela. Pybnn.
2020.
Sijia Liu, Parikshit Ram, Deepak Vijaykeerthy, Djallel Bouneffouf, Gregory Bramble, Horst Samu- lowitz, Dakuo Wang, Andrew Conn, and Alexander G. Gray. An ADMM based framework for automl pipeline configuration. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAÍ 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, FAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 4892 4899. AAAI Press, 2020. URL https: //tp::://js..aat aaai. org/ index..ix.. php/ http///AAaaaatee/eeeeee
Luke Metz, Niru Maheswaranathan, Ruoxi Sun, C. Daniel Freeman, Ben Poole, and Jascha Sohl- Dickstein. Using a thousand optimization tasks to learn hyperparameter search strategies. CORR, abs/2002. 11887, 2020.
Felix Mohr, Marcel Wever, and Eyke Hüllermeier. MI-plan: Automated machine learning via hierar- chical planning. Mach. Learn. 1997 2018, doi: 10.10077/109944--18- URLhtttsst http//ii.....o////oooooooo 11t.//00...000111111111 httppp/s../opooooo.//o..///00/00

---

Under review as a conference paper at ICLR 2023
Randal S. Olson and Jason H. Moore. TPOT: A tree-based pipeline optimization tool for automating machine learning. In Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.), Proceedings of the 2016 Workshop on Automatic Machine Learning. AutoML 2016, co-located with 33rd International Conference on Machine Learning (ICMI. 2016), New York City, NY, USA, June 24, 2016, volume 64 of JMLR Workshop and Conference Proceedings, pp. 66 74. JMLR.org. 2016. URL
m.r.. ott. tpot 2016. 20t
Ekrem Ozturk, Fábio Ferreira, Hadi Samer Jomaa, Lars Sehmidt-Thiemns Josif Grabocka, and Frank Hutter. Zero-shot automl with pretrained models. In International Conference on Machine Learning (ICML). 2022.
Massimiliano Patacchiola, Jack Turner, Elliot J Crowley, Michael O'Boyle, and Amos J Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. Advances Neural Information Processing Systems, 13:16108-16118 2020.
Valerio Perrone, Rodolphe Jenatton, Matthias W. Seeger, and Cédric Archambeau. Scalable hyper- parameter transfer learning. In Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 6846-6856, 2018
Herilalaina Rakotoarison, Marc Schoenauer, and Michèle Sebag, Automated machine learning with monte-carlo tree search. In Sarit Kraus (ed.), Proceedings of the Twenty-Eighth International
Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 3296-3303 ijcai.org, 2019. htRpsttp::/ :ttps://ddoo /tw.///o..ooo// orw..o...g1..oo 2ttp:/1....99///111113 2019 /47
Carl Edward Rasmussen and Christopher K. Williams. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press, 2006. ISBN 026218253X.
Nicolas Schilling, Martin Wistuba, Lucas Drumond, and Lars Schmidt-Thieme. Joint model choice and hyperparameter optimization with factorized multilayer perceptrons. In 27th IEEE Interna- tional Conference Tools with Artificial Intelligence, ICTAI 2015, Vietri sul Mare, Italy, November 9-11, 2015, pp. 72-79. IEEE Computer Society, 2015. URL https: //oo. wwtp:...../...0.///...0..... ICTAI 2015. 24
Rishit Sheth. pmf-automl. httpppps///.ttaa......////ttttts
2018.
Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pp. 2960-2968 2012.
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Md. Mostofa Ali Patwary, Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 2171-211111 2015.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5- 5-10, 2016, Barcelona, Spain, pp. 4134-4142, 2016.
Thomas Swearingen, Will Drevo, Bennett Cyphers, Alfredo Cuesta-Infante, Arun Ross, and Kalyan Veeramachaneni. ATM: A distributed, collaborative, scalable system for automated machine learning. In 2017 IEEE International Conference on Big Data (IEEE BigData 2017), Boston, MA, USA, December / 11-14, 20/7, pp. 152-162. IEEE Computer Society, 2017. doi: 10. 109/BigData.. 2017.8257923. URL https::/:/// /oii wtt./////000./.. 1009/ 1tt.11100////aaaa..aa 2017. 8257923
Chris Thorton, Frank Hutter, Holger H. Hoos, and Kevin seyton-Brown. Auko-weka: Automated selection and syper--arrmeter optimization of classification algorithms. CoRR, 120..199,,719, 2012.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
12

---

Under review as a conference paper at ICLR 2023
Ying Wei, Peilin Zhao, and Junzhou Huang. Meta-learning hyperparameter performance prediction with neural processes. In International Conference on Machine Learning, pp. 11058-11067. PMLR. 2021a.
Ying Wei, Peilin Zhao, and Junzhou Huang. Meta-learning hyperparameter performance prediction with neural processes. In International Conference on Machine Learning, pp. 11088-11067. PMLR, 2021b.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kemnel learning. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 370-378, Cadiz, Spain, 09-11 May 2016. PMLR.
Martin Wistuba and Josif Grabocka. Few-shot bayesian optimization with deep kernel surrogates. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Chengrun Yang, Yuji Akimoto, Dae Won Kim, and Madeleine Udell. OBOE: collaborative filtering for automl model selection. In Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (eds.), Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019, pp. 1173-1183. ACM, 2019. URL htllhttps://dde
orq 10. 11../129225200 3330909
Chengrun Yang. Jicong Fan, Ziyang Wu, and Madeleine Udell. Automl pipeline selection: Efficiently navigating the combinatorial space. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (eds.), KDD 20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Virtual Event, CA, USA, August 23-27, 2020, pp. 1446-1456. ACM, 2020. URL ht https: doi. wtt.:..........3.22221..1 340319191744 3403197
13

---

Under review as a conference paper at ICLR 2023
A POTENTIAL NEGATIVE SOCIETAL IMPACTS
The meta-training is the most demanding computational step, thus it can incur in high energy consumption. Additionally, DeepPipe does not handle fairness, it may find pipelines that biased by the data.
B LICENCE CLARIFICATION
The results of this work (code, data) are under license BSD-3-Clause license. Both PMF dataset Sheth (2018) and Tensor-OBOE dataset dataset/Akimoto Akimoto & Yang 2020) hold the same license.
C DISCUSSION ON NUMBER OF EVALUATED PIPELINES
Based results from Experiment 3, we report the average (and stan- dard deviation) of the number of ob- served pipelines among all the com- pared methods in 10 and 60 minutes on Table 3 This is an important metric understand the optimization overhead introduced by the method.
Table 3: Average Number of Observed Pipelines on OpenML Datasets
Method
10 Mins
1 Hour
TPOT
45.48 46.25
+ 4 41.67
Tensor-OBOE OBOE
84.43 57.61
178.95 69.04
OBOE
120.35
467.09 330.34 330.34
For instance, a method that explores few pipelines during fixed time win- dow, might use expensive computa- tions during the pipeline optimization. We notice that DeepPipe achieves the best results (see Table I by using a reasonable amount of pipelines, i.e. the optimization overhead introduced our method small compared to other approaches such as TPOT and OBOE.
SMAC
80.76 + 115.04
452.35 + 637.08
PMF
126.37 + 197.61
523.71 + 663.07
DeepPipe
94.51 + 128.62
356.71 + 379.62
D DISCUSSION ON THE INTERACTIONS AMONG COMPONENTS
The encoder and aggregation layers capture interactions among the pipeline components, and therefore are important to attain good per- formance. These interactions are reflected in the features extracted by these layers, i.e. the pipelines representations obtained by DeepPipe. These representations lie metric space that captures relevant information about the pipelines and which can be used the kernel for the Gaussian Process. Using the original input space does not allow extract rich representations. To test this idea, we meta-train four version of DeepPipe with and without encoder and aggre- gation layers on our TensorOBOE meta-train and then on the meta-test split. In Fig- u we show that the best version is obtained when using both encoder (Enc.) and aggregation (Agg.) layers (green line), whereas the worst ver- sion is obtained when using the original input space, i.e. encoder and aggregation layers. Having encoder helps more than not having en- coder, thus it is important to capture interactions among hyperparameters in the same stage. As having an aggregation layer is better than not, is important to capture interactions among components from different stages.
Interactions Ablation
3.0
2.8
2.6
2.4
2.2
5
20
40
60
80
100
No. of Explored Pipelines
1-Fne (0-Agg
Fnce/1-Agg.
2-Ene. 2-Ene,/0-Age /0-Agg
O-Ene./O-Agg,
Average rank for DeepPipe with and without encoder and and aggregation layers.
Figure 5:
14

---

Under review as a conference paper at ICLR 2023
Tea
PMF
ZAP
1.00
1.0
1.55
1.50
1.4
1.45
1.40
20
00
SO
100
5
20
40
GO
80
100
20
40
60
80
100
No. of Explored Pipelines
No. of Explored Pipelines
No. of Explored Pipelines.
DeeplPipe
SKL
Figure 6: Comparison with Structured Kernel Learning (SKL)
Table 4: Comparison with AutoPrognosis
Eво
Method
Avg. Rank
Std. Rank
Avg. Acc.
Std. Acc.
Avg Time
Std. Time
AutoProg.
1.5588
0.4416
0.8637
0.1143
19324
12934
50
DeepPipe
1,4411
0.4416
0.8692
0.1113
903
1548.
AutoProg.
1.5133
0.4694
0.8715
0.0949
I 18502
11176
100
DeepPipe
1.4866
0.4694
0.8727
0.0972
2221
5405
E COMPARISON WITH STRUCTURED KERNEL LEARNING (SKL) AND AUTOPROGNOSIS
AutoPrognosis (Alaa & van der Schaar 2018) uses Structured Kernel Learning (SKL) and meta- learning to account for the interactions among the pipelines components. SKL decomposes the original input space by making up N group of pipelines components, e.g. Random Forest and SVM in group separated from Linear Regression and Logistic Regression. The hyperparameters group of pipelines components is then passed through a kernel, and then the .N resulting kernels added. This effectively builds up kernel with additive structure (Gardner et al., 2017), however they not using feature extractor like DeepPipe. We compare SKI. against noneprrtrddeed DeepPipe on Figure 6 on three meta-datasets, where is noticeable that our method outperforms this strategy.
every
Additionally compare DeepPipe with the whole algorithm introduced by Auto AutoPrognosis 2.0 [Imrie] et al. 2022) on the Open ML datasets for 50 and 100 BO iterations (EBо). We report the average and standard deviation for the rank, accuracy and time. DeepPipe achieves the best average rank, ie. lower average rank than AutoPrognosis. This is complemented with the having the highest average accuracy, Interestingly, our method approximately one order of magnitude faster than AutoPrognosis, We notice this is due to the time overhead introduced by the Gibbs sampling strategy for optimizing the structured kernel, whereas our approach uses gradient- based optimization.
Experimental Set-Up for DeepPipe. For our comparison with SKL, we use the same hyperparame- ters and architecture as for the Experiment 1. When comparing with AutoPrognosis, we use the same hyperparmeters and architecture as for the Experiment 2, pre-trained on the Tensor-OBOE meta-train split.
Experimental Set-Up for SKL and AutoPrognosis For SKL used the default strategy with N (Alaa & van der Schaar 2018). For AutoPrognosis, we use the implementation in the respective author's 's repository We ran with the default configuration, but limited the search space of classifiers to match the classifiers on the Tensor-OBOE meta-dataset
https: / / thub. com ahmedma Aut.//oooorooooiiggg
'Specifically, the list of classifiers is: Random Forest, Extra Tree Classifier, Gradient Boosting". Logist Regression, MLP, linear SVM, kNN, Decision Trees, Adaboost, Bernoulii Naive Bayes, Gaussian Naive Bayes, Perceptron.
15

---

Under review as a conference paper at ICLR 2023
F DISCUSSION ON THE INDUCTIVE BIAS vs. PRE-TRAINING EFFECT
How shallow/deep should the encoder networks be compared to the aggregation network? We hypothesize that deeper encoders help in the transfer-learing setup where there exist only a few evaluated pipeline configurations on past datasets. To test this hypothesis, we assess the performance of DeepPipe with different network sizes and meta-trained with different percentages of meta-training tasks: 0.5%, 1% 5%. 10%, 50%, and 100%. As we use the Tensor-OBOE meta-dataset, this effectively means that we use 16, 33, 165, and 330 tasks respectively.
The results of these experiments are shown in Figure 7 Here we ablate DeepPipe with different numbers of encoder layers while pre-training on different fractions of the meta-training tasks. We ran the experiment for three values of F. The presented scores
F-4
F-6
F-8
0.5%
2.05
2.08
1.88
2.0
1.89
2.1
2.12
2.02
1.86
Fraction of Meta-Taai Data
1%
1.96
2.18
1.86
2.11
1.9
1.99
2.08
1.91
2.01
1.88
2.02
2.09
2.01
1.95
2.04
1.98
2.07
1.96
are the average ranks among the three DeepPipe configurations (row-wise). The average rank is computed across all the meta-test tasks and across 100 BO iterations. The results indicate that deeper encoders achieve bet- ter performance when a small num- ber of meetattriningg tasks is available. In contrast, shallower encoders are needed if more meta-training tasks are available. Apparently the deep aggre- gation layers & already capture the in- teraction between the hyperparameter configurations across algorithms when large meta-dataset of evaluated pipelines is given. The smaller the meta-data of evaluated pipeline configurations, the more inductive bias we need to implant in the form of per-algorithn encoders.
10%
1.91
2.03
2.06
1.87
2.07
2.06
1.91
1.93
2.16
1.9
2.06
2.04
1.94
2.03
2.03
1.87
2.04
100%
1.81
2.1
2.09
1.89
1.98
2.13
1.86
2.16
1.98
0
Encoder Depth
Figure 7: Comparison of the average rank for DeepPipe with different number encoders under different percentages of meta-train data. The total number of layers is always the same.
G ADDITIONAL INFORMATION ON EXPERIMENTAL SET-UP
In all experiments (except Experiment 1), we meta-train the surrogate following Algorithm iin Appendix [ for 10000 epochs with the Adam optimizer and a learning rate of 10 batch size 1000, and the Matérn Kernel. During meta-testing, when we perform BO to search for pipeline, we fine-tune only the kernel parameters y for 100 gradient steps. In the non non-transfer experiments (Experiment 1) we The rest the training settings are similar to the transfer experiments. In Experiment we fine- tune the whole network for 100 steps when no encoders the encoder associated with the omitted estimator and freeze the rest of the network. We ran all experiments on CPU cluster, where each node contains two Intel Xeon E5-2630v4 CPUs with 20 CPU cores each, running at 2.2 GHz. We reserved a total maximum memory of 16GB. We discuss how we implemented DeepPipe efficiently We associate algorithms with no hyperparameters to the same encoder. We found that adding the mme-Hot-Encoding of the selected algorithms per stage as an additional input helpful. Therefore, the input dimensionality of the aggregated layers is equal to the dimension after concatenating the encoders output F. [,Q, M,). Further details on the architectures for each search space are specified in Appendix M Finally, use the Expected Improvement as acquisition function for DeepPipe and all the baselines.
architecture with F = 8 and fine-tuned the network for 10000 iterations.
used. Otherwise,
fine-tune only
MLP with masked layers in Appendix N
Initial Configurations For the experiments with the PMF-Dataset, choose these configurations with the same procedure as the authors Fusi et al.) (001)), where they use dataset meta-features to find the most similar auxiliary task to initialize the search the test task. Since do not have meta-features for the Tensor-OBOE meta-dataset, we follow greedy initialization approach Metz al. 2020). This was also applied to the ZAP-Dataset. Specifically, we select the best-performing
'We make our code available in https:://aannneeee 4open. wcciencc/r//eepee
9K
16

---

Under review as a conference paper at ICLR 2023
pipeline configuration by ranking their performances the meta-training tasks. Subsequently, iteratively choose four additional configurations that minimize LigTasks ft. where fg = minpeX Tt.p- given that Pt is the rank of the pipeline p on task t.
H ADDITIONAL RELATED WORK
Hyperparameter Optimization (HPO) has been well studied over the past decade (Bergstra & Bengio 2012). Techniques relying Bayesian Optimization (BO) employ surrogates approximate
the response function of Machine Learning models, such as Gaussian Processes (Snoek et al. (0122)
Random Forests (Bergstra et al. 2011) Bayesian Neural Networks (Snoek
2015 | (Srringenberg
et al. 2016). Further improvements have been achieved by applying transfer learning, where existing evaluations auxiliary tasks help pre-training or meta-learning the surrogate. In this sense, some
approaches use pre- tre-trained neural networks with uncertainty outputs (Wistuba & Grabocka [2021
Perrone al., 2018;
Wei al. 2021b), or ensembles of Gaussian Processes (Feurer
2018).
Deep Kernels propose combining the benefits of stochastic processes such as Gaussian Processes with neural networks (Calandra et al.] 2016; Garnelo al. 2018 Wilson et al. 2016). Follow-up work has applied this combination for training sho-shot classifiers (Patacchiola et al. 2020). In the
area of Hyperparameter Optimization, (Snoek et al. 2015) achieved success on BO by modeling the output layer of deep neural network with a Bayesian linear regression. |Perrone et al. (2018) extended this work by pre-training the Bayesian network with auxiliary tasks. Recent work proposed to use non-linear kernels, such as the Matérn kernel, on top of the pre-trained network to improve the performance of BO (Wistuba & Grabocka 2021
Wei et al. 2021a).
I BAYESIAN OPTIMIZATION (BO)
In BO we fit surrogate iteratively using the observed configurations and their response. Posteriorly, its probabilistic output is used to query the next configuration to evaluate (observe) by maximizing an acquisition function. A common choice for the acquisition is Expected Improvement, defined as:
Epx (H) = E [max (u(px) - ymmx 0}]
where Vmax is the largest observed response in the history H and μe is the posterior of the mean predicted performance given by the surrogate, computed using Equation A common choice as surrogate is Gaussian Process, but for Pipeline Optimization we introduce DeepPipe.
J DeepPipe META-TRAINIM
(C.) ((xx(t..y((()))) ((.)))..... (.90), (t..2))). t (1...., T} Given a task with observations 74 = {px (t.1) the objective function to minimize can be derived from the negative log marginal likelihood from the Gaussian Process p(H.)) (O, where (()) is the covariance matrix induced by DeepPipe with parameters 0, 7. Specifically, the negative log marginal likelihood (Rasmussen & Williams/ 2006
log (H.) log (9) -(t (9)00)) (0,7)~1y()) + log (0,)))
The Equation 5 the multi-task objective function that involves all the meta-learning tasks with indices / T}.
We use auxiliary tasks to learn a good initialization for the surrogate. We sample batches from the meta-training tasks, and make gradient steps that maximize the marginal log- ligeliillhoodd in Equation 5) similar to previous work (Wistuba Grabockal 202I). The training algorithm for the surrogate is detailed in Algorithm| [ Additionally, we apply Early Convergence by monitoring the performance on the validation meta-dataset. Every epoch, we perform the following operations for every task e I...T:: i) Draw set of observations (pipeline configuration and performance), ti) Compute the negative log marginal likelihood (our loss function) as in Equation / ii compute gradient of the loss with respect to the DeepPipe parameters and iv) updated DeepPipe parameters.
17

---

end Under review as a conference paper at ICLR 2023
Algorithm 1: DeepPipe Meta-Training
Input: Learning rates 77, meta-training data with I tasks H Ut=1.TT H(t, number of epochs E, batch size
Output: Parameters and
Initialize w and at random;
for E times do
for 1 E T) do
Sample batch B = ((p (t.3) .(())) H(:
Compute negative log-likelihood L on B. (Objective Function in Equation 5)
(ass 0ags 1/V gana
gene + Hone_ 11 V L;
7 N.
end
Algorithm 2: Bayesian Optimization (BO) with DeepPipe
Input: Learning rate initial observations H {(p)) y )=mm)......... I...... pretrained surrogate with
parameters 0 and 7, number surrogate updates Erest, BO iterations ERO, search
space of pipelines P
Output: Pipeline Configuration PA
Function FineTune (H, 7, 1) Frest):
for Frest times do Compute negative log- -iiiliikehhdd L on D. (Objective function in Equation 5 with T 1); CV
end return 7
F Function BO (H. 17, 0, 7, Erest: EBO):
for EBO times do
7' FFFneTTuneH, y, 17, Erest);
Compute P E arg max, EXEP El(p,, 7 ()
10
Observe performance y of pipeline p
11
Add new observation H HU (()))'))}
13
end
14
Compute best pipeline index i, E arg max [7.. V:
15
return X
K
DeepPipe META-TESTING
When a new pipeline is to be optimized on new dataset (task), we apply BO (see Algorithm Every iteration we update the surrogate by fine-tuning the kernel parameters. However, the parameters of the MLP layers can be also optimized, as we did on the Experiment 1, in which case the parameters were randomly initialized.
L ADDITIONAL RESULTS
In this section, present further results. Firstly, show ablation of the factor that determines the number hidden units (F)) in Figure It shows that F' 8 attains the best performance after exploring 100 pipelines in both datasets. Additionally, present the average regret for the ablation of F, and the results of Experiment and in Figures
10 and respectively.
Table present the extended results of omitting estimators in the PMF Dataset. From these, draw the same conclusion as in the same paper: having encoders help to obtain better performance when a new algorithm is added pipeline.
18

---

Under review as a conference paper at ICLR 2023
Tensor-OBOK Meta-Datasel
PMF Mota-Dataset
3.5
Average Rank
3.2
F=2
F=4
F=6
3.0
F=8
2.8
F=10
2.5
2.6
20
40
60
80
100
5
20
40
60
80
100
No. of Explored Pipelines
No. of Explored Pipelines
Figure 8: Comparison of different F values in DeepPipe (Rank).
Tensor-ODEE Data-Datsstt
PMF Meta-Dataser
0.30
Average Regret
0.04
0.25
F=2
F=4
0.03
0.20
R=6
F=8
0.02
0.15
F=10
0.10
0.01
20
40
60
100
40
100
Explored Pipelines
Explored Pipelines
Figure Comparison of different F values in DeepPipe (Regret).
Table 5: Average rank among DeepPipe variants for newly-added algorithms (PMF)
Omitted in
Omitted Estimator
Enc.
MTd.
MTr.
MIc.
ET
RF
XGBT
KNN
GB
DT
Q/LDA
NB
/
3.1527
3.1645
3.2109
3.2541
3,2874
3.2741
3.1911
3,0263
x
/
X
3.2462
3.3208
3,2592
3.3180
3.2376
3.2249
3.3557
3,3993
2.5996
2.5947
2.5664
V
x
3.0464
2.8550
3.0850
2,8845
2.9397
3.0316
2.9530
3,0596
x
2.9838
3.0601
3.0439
2.9486
2,9051
2.9029
2.8750
2,8934
We carry out an ablation to understand the difference between the versions of Deep Pipe with/without encoder and with/without wansfer-learning using ZAPP Meta-dataset. As shown Figure 12. the version with transfer learning and one encoder performs the best, thus, highlighting the importance of encoders in transfer learning our DeepPipe surrogate.
M ARCHITECTURE DETAILS
The input to the kernel has dimensionality of -=20. We fix it, to be the same the output dimension for PMFs. The number of neurons per layer, as mentioned in the main paper, depends on F. Consider architecture with with encoder layers and f. aggregation layers, and hyperparameters Arj, i E [1.. E (1...M,} (following the notation section 4.11) with Q: = max, [Ar, then the number of weights (omitting biases for the sake of simplicity) will be:
(F.
F. [0.
Σ | 1
+ (la -1)
(8)
19

---

Under review as a conference paper at ICLR 2023
Tensor OBOE Meta Dataved
PMF Meta Datasel
ZAP Meta Dataeet.
0.005
Average Regret
0.04
0.004
0.03
0.20
0.003
0.02
0.15
0.002
5
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
No. of Explored Pipelines
No. of Explored Pipelines
No.
Explored Pipelines
DeepPipe
RS
GP
DNGO
SMAC
Figure 10: Comparison of DeepPipe vs. non transfer-learning PO methods in Experiment 1 (Regret)
Temsor: OBOO Geta-Datased
PMF Meta- Dataset
ZAP Meta. Dataset
Average Regret
0.04
0.25
0.004
0.03
0.20
0.02
0.15
0.002
0.01
0.10
5
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
No. of Explored Pipelines
No. of Explored Pipelines
No. of Explored Pipolines
DeepPipe PMP
T-OBOE
FMLP
RS
OBOE
RGPE
Figure 11: Comparison of Regret DeepPipe vs. transfer-leaming PO methods in Experiment (Regret)
the architecture has l, encoder layers and l, aggregation layers, then number weights is given by:
(le Me (F (F-Q.)) tlo F EQ: 2
E Ansl ((e-)) Eme
Ablation Encoders
Ablation of Transfer
2.4
2.75
Avergge Rank
Average Rank
2.50
2.0
2.25
1.8
2.00
1.6
5
40
80
100
20
40
60
100
No. of Explored Pipelines
No. of Explored Pipelines
One Encoder
RS
DeepPipe Transfer
GP
No Encoder
DeepPipe Non-Transfer
RS
Figure 12: Ablations the ZAP meta-dataset
20

---

Under review as a conference paper at ICLR 2023
Tensor-OBOE
PMF
No. of Weights (Log)
105
Encoder layers
0
10
1
2
104
4
6
10
2
6
10
F
F
Figure 13: Number of weights in the MLP for given value of F and encoder layers.
other words, the aggregation layers have F Σ: Qi hidden neurons, whereas every encoder from the i-th stage has F Q; neurons per layer. The input sizes are Ex, |Acy and |As,g for both cases respectively. The specific values for |Ai,y and Qi per search space are specified in Appendix R
In the search space for PMF, we group the algorithms related to Naive Bayers (MultinomialNNB BernoulliNB, NaussianNB) in single encoder. In this search space, also group LDA and QDA. In the search space of TensorOboe, we group GaussianNB and Perceptron as they do not have hyperparameters. Given these considerations, can compute the input size and the weights per search space as function of Eay Le, F as follows:
(i) Input size:
Input size (PMF)
Aid 72
1.
# Input (TensorOboe) =
Σ
| = 37
(10)
12
Input (ZAP) =
A 35
(ii) Number of weights for architecture without encoder layers:
Weights (PMF) 720 F 256 (la 1) F2
# Weights (TensorOboe) 444 F + 144 (l, - 1) F22
(11)
# Weights (ZAP) = 1085 + 961 (fa - 1) F22
(iii Number of weights for architecture with encoder layers:
Weights (PMF) 886 (1376 (Le - 1) + 256 (.) F2
# Weights (TensorOboe) = 161 F + (271 (fe - 1) + 144 l,) F2
(12)
# Weights (ZAP) = - (965 (Le -1) + 969 -2)) FF
According the previous formulations, Figure shows how many parameters (only weights) the MLP has given a specific value of and of encoder layers. We fix the total number of layers to four. Notice that the difference in the number of parameters between architecture with and 2 encoder layers is small in both search spaces.
N COMPUTATIONAL IMPLEMENTATION
DeepPipe's architecture (encoder layers aggregated layers) can be formulated as a Multilayer Perceptron (MIP) comprising three parts (Figure 14). The first part of the network that builds the
21

---

Under review as a conference paper at ICLR 2023
Selection and Concatenation
Inpur Layer
Encoder Layer
Aggregation Layer
(D)
(2)
1,1
(1)
1,2
x
(2)
2,2
Learneable Weights
Weights set to one
Weights set zero
Figure 14: Example of the Implementation of DeepPipe as MLP. (AA) indicates the k-th hyperparam- eter of the j-th algorithm in the i-th stage. In this architecture, the first stage has two algorithms, thus two encoders. The algorithm is active for stage The second stage has only one algorithm.
layers with encoders is implemented layer with masked weights. We connect the input values corresponding to the hyperparameters A(E))) of the j-th algorithm of the i-th stage to a fraction of the neurons the following layer, what builds the encoder. The fraction of neurons, as explained in section 5.4 is F max; D- The rest of the connections are dropped. The second part is layer that selects the output of the encoders associated with the active algorithms (one per stage), and concatenates their outputs (Selection & Concatenation). The layer's connections to one or zero during forward and backward pass. Specifically, they are one if they are connecting outputs of encoders of active algorithms, and zero otherwise. The last part, aggregation layer, is fully connected layer that lear interactions between the concatenated output of the encoders. By implementing the architecture in Py Torch), faster forward and backward passes are obtained. We only need to specify the selected algorithms in the forward-pass. so that the weights in the Encoder Layer are masked and the ones in the Selection & Concatenation are accordingly set. After this implementation, notice that DeepPipe a MLP with sparse connections.
fixed be either
MLP instead of multiplexed list of nodes (e.g. with module list
O VISUALIZING THE LEARNT REPRESENTATIONS
We train a DeepPipe with 2-layer encoders, 2 aggregation layers, 20 output size and F = 8. To visualize the pipelines embeddings, we apply TSNE (T-distributed Stochastic Neighbor Embedding). As plotted in Figure 15] the pipelines with the same estimator and dimensionality reducer are creating clusters. The groups this latent space are also indicators of the performance on specific task. In Figure 16 we show the same embeddings of the pipelines with color marker indicating its accuracy on two meta-testing tasks. Top-performing pipelines (yellow color) are relatively close to each other in both tasks, building up regions of good performing pipelines. These groups of good pipelines are different in both cases, which indicates that there is not single pipeline that works for all tasks. DeepPipe maps the pipelines to embedding space where is easier to assess the similarity between pipelines and to search for good-pent good-performing pipelines. However, the type of pipeline (good performing pipelines, bad performing pipelines) depends on the task.
22

---

Under review as a conference paper at ICLR 2023
100
100
75
75
50
25
0
-55
-25
-50
-75
-75
-100
-100
100
-50
0
50
100
-100
-50
50
100
ExtraTrees
RF
DT
PCA
VarianceThreeshold
GBT
ISVM
AB
SelectKBest
Logit:
KNN
GN
MLP
Figure 15: Learnt representations in dimensions for estimators (left) and dimensionality reducers (right) from the Tensor-OBOE meta-dataset.
100
100
75
75
25
25
25
-25
-50
-50
-75
-100
-100
-100
-50
50
100
-100
-50
50
100
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
0.2
0.3
0.4
0.5
0.6
0.7
Figure 16: Learnt representations for two tasks with different accuracy levels.
23

---

Under review as a conference paper at ICLR 2023
P META-DATASET PREPROCESSING
We obtained the raw data for the meta-datasets from the raw repositories of PMF and TensorQBOE PMF repo provides an accuracy matrix, while Tensor-OBOE specifies the error. We take the pipelines configurations and concatenate the hyperparameters in both meta-datasets. Then we proceed with the following steps: One-Hot encode the categorical hyperparameters, 2) apply a log transformation Inew In(x) to the hyperparameters whose value is greater than 3 standard deviations, 3) scale all values to in the range [0,1]. The variables coming from categorical hyperparameters are named
original-variable-mann samiiabbl--mm_aaaeeeeae
Q ABBREVIATIONS
(i) Abbreviations in Table 2:
ET: ExtraTrees, GBT: Gradient Boosting, 3) Logit: Logistict Regression 4) MLP: Multilayer Perceptron 5) RF: Random Forest, 6) ISVM: Linear Support Vector Machine, 7) kNN: k Nearest
Neighbours, 8) DT. Decision Trees, 9) AB: AdaBoost, 10) GB/PE= Gaussian Naive Bayes/Perceptron.
(ii) Abbreviations in Table 3:
ET: Extra'Trees, 2) RF: Random Forest, XGBT: Extreme Gradient Boosting, kNN: K-Nearest Neighbours, 5) GB: Gradient Boosting, 6) DT: Decision Trees, 7) Q/LDA: Quadratic Discriminant
Analysis/ Linear Discriminant Analysis, 8) NB: Naive Bayes.
R META-DATASET SEARCH SPACES
We detail the search spaces composition in Tables 6 and 7 hyperparameters, number of components per stage M, the number of hyperparameters per algorithm [' and the maximum number of hyperparameters found in an algorithm per stage Q; For the ZAP meta-dataset, defined pipeline with two stages: (i) Architecture, which specifies the type or architecture used (i.e. ResNet18, EfficientNer-Bo EfficientNet-B3 EfficientNet-B22 and (ii) Optimization-relatte Hyperparameters that are shared all the architectures.
We specify the stages, algorithms,
"https://ithu..com//// http://hhh...cmm//cccchSh/hmaaam
24

---

Under review as a conference paper at ICLR 2023
Table 6: Search Space for PMF Meta Dataset
Stage
Qi
M;
Algorithm
|Aa.5l
Hyperparameters
Polynomial
3
include bias, interaction only, degree
Preprocessor
2
PCA
2
keep variance, whiten
bootstrap, min sampues leal, n. essimmators, max leaturess weight fraction, Jeaf, min saonssamplesssgg max, max_depit depth
ExtraTrrees
9
bootstrap, min sampless Jeal, n, estimators, max, Jeatures. min, weight, fraction leaf, min samples split, depth, criterion_entropy entropy, criterion gini
RandomForest
Estimator
13
8
rey alpha, col, sample bytree, colsample bylevel, scale pos, weight, leaming rate,
XgradiennBoosding
13
della, step, base
essimatoos, subsample,
reg. lambdda, min child weight, max depth, gamma
KNN
neighbors, weights distance, weightx uniform
max leaf nodes, learning rote, min samples leaf.
GradienBBossing
10
n. estimatoos,, subsample, min, weight Fraction Jeaf, meatuuus min samples split, max depth, loss, deviance
max, leaf nodes, samplex, leaf, min_mmnnmppppa samples, split, max depth, spliteer_ese best, criterion oriterion_entropy. criterion _iniii
Jeatures,
Tree
min weight fraction leaf, min
shrinkage factor, n components, tol, shrinkage_-1,
LDA
shrinkage auto, shrinkage, manual
QDA
reg_ param
BemoulliNB
2 alpha, fit prir
MultinmialNN
2 alpha, fit fipp
GaussiaNB
1
upply_gaussian
Table 7: Search Space for Tensor-OBOE Meta- Dataset
Stage
Qi
M;
Algorithm
|Ac.5l
Hyperparameters
Imputer
4
Simplelmputer
Stratgy constant, Strategy Beaiiogy_menn.ee haa__mmann.Sttt Strargyy _median. Stratogy_nne htt__most__fooo frequuuuu
Encoder
I
OncHotEncoder
Handle, Bockll___knoww____ _gnooe
Scaler
StandardSealer
N_components
Dim. Reducer
BeleetKBest
Thrihoce
ExtraTrees
3
min samples sppnp criterion, entropy, criterion gini
learning rate, max depth, maax_feature__Neea max, feeatures_oog Jog2
Gradient Boosting
Logit
5
C. penalty Jl. penalty 12, sovler_ liblinear, solver_saga
Estimator
5
10
alpha, leaming_rake Bartninng_rtt_iiiee init leaminn_rrace. sad_apptttt__eiive....o__a solver adam, solver sgd
Randorn Forest
3
min samples split, criterion, entrroyyy criterion.gin
ISVM
1
KNN
2
n_ncighbors,
Decision
min fann_sample__sve split
AdaBoust
2
learning_rad rale, n, estimatoes
GaussianNB
1
Perceptron
25

---

Under review as a conference paper at ICLR 2023
Table 8: Search Space for ZAP Meta-Dataset
Stage
Q:
Mr
Algorithm
|Ar,y
Hy perparameters
ReaNet
IsActive
Architecture
EfficientNetBOO
IsActive
EthicienNee-B11
IsActive
EfficienNNt-B22
IsActive
early eepcch, first simpsimmpe__mee
max, inner_loop___ har__looo__rrrta ratio.
skip_valid scorv chreshhld test_afeer_ slss test after__it least seconds, max.
test after at least_ scconds step. batch_size, cv_ ov_valid_rate ratio, max size, mux, validalii__a count, steps, perpppper_aeee per__err____ per_ep trinn info, samplee
Common Hlperparameeers 31
optimizer.amsgrad optimizer freeze partion, optimizerli.
optimizer.min. Jt,
optimizer.esstroor,
optimizer warm up_opoch.
warmup_ multiplier. optimizer.wd
simple_model model LR, simple m @ampl__mdel__Ee RE: simple
SVC, optimizerschedulee cosine,
optimizer.type. seaniizsttye_Adam.. sptimizer.lpe__Wie Addmm
THEORETICAL INSIGHT OF HYPOTHESIS 5
Here, we formally demonstrate that the DeepPipe with encoder layers is grouping hyperparameters from the algorithm in the latent space, better than DeepPipe without encoders, formulated on Corollary S.4 which is supported by Proposition S.3
Lemma S.1. Given w E RM, vector of weights with independent and identically distributed components w; E [wi,..., was) such that wi p(w), the expected value of the square of the norm Ep(w; (2) is given by M (Me + o where pwu and ow are the mean and standard deviation of p(w) respectively.
Proof.
M
E Ep(m) (1|w|12) = Ep((c)
(13)
M
E
Ep(wr) Ep(n)((??)
(14)
1=1
Σ i=1
Men + o
(15)
M
(u2 + +2)
(16)
L
Lemma S.2. Consider a linear function with scalar output = w x where w E Rx is the vector of weights with components w;, E (I....,M. M), a Mx1 are the input features. Moreover, p(w). The expected value of
consider the weights are independently and identically distributed the norm of the output is given by Eyp(w) (ooo)=11))) 12) (1)))=x)11)- (n2 + +oo) ||==| 22 Die -i=1 v4.
26

---

Under review as a conference paper at ICLR 2023
Proof.
M
Bp(w) (ww²x))3 Ep(w)
Σ
Ws Dz
(17)
M
i-1
Ep(w)
W - F3g +
Wi + W/ 33 . 23
(18)
j
M
M f-1
Ep(1) (w .2, + 2
B pl)) (wy wj) 24 2z
(19)
1= 1=1
(20)
Since wi, wy are independent then Eptw) (w; + w)) = Bp(w) (w;) (Bj())(w)) - n Moreover, with
slight abuse in notation, we denote Σ M
Di=1 Di=1 2; Tj x. Given lemma S.1
obtain:
Ep(w) (ww"x) x)²) (m² + +w)) || 2 2 mlw x 2 x = Dw
(21)
(22)
where ( is introduced as operation to simplify the notation.
Proposition S.3. Consider two vectors x', E RM,, and two weight vectors ib and w',
TT R. nh x E R, such that the weights are iid. Then Ep(w)) (ww6 w x')?) Ep(w) ((w x o²'a))))
Proof. Using lemma S.2 and decomposition the argument within square:
Ep(w)((w" x w 20)3) Ep(w) (ww a) (w' x²)² -2 2 20 2 w x')
(23)
Dw(() + (x') - 2 Bp(w) (w a wwww"" x')
(24)
M
M' Σ
Dw(() + (wx) - 2 Ep(w)
W
21
wjy xzz
(25)
i=1
3=1
M
M'
Dw(() + Dw(x') -2 -2 Ep(w))
Wi'
=.."" - +x)
(26)
D+) + (m' ((((aa))))) 2- Σ Σ Ep(nn) (woy (.,) - xzy' - (27)
Since ip and w' are independent, then ( w;) = Ep(w) (wy') Ep(w) = 2 Thus,
A
M
Ep(w) (1w² a w
( =')) (w) + Dm - 22 1
(28)
When computing Ep(w) (iw ãt ut xd)), that the weights not independent, thus
Ep(()) (w); w))) 2 + 2 and
27

---

Under review as a conference paper at ICLR 2023
M
M
Bp(w) ((v) it - --="=""a" xb))) = (x) + (w' - 2. (n + +
J/3
5z
(29)
i=1
M M
Dw + Dz)()') (oocaa))- - pw
3
(30)
i=1 j=1
< Bp(w)) (lw xc 10--w n2 10'=²''² x2)²)
(31)
Corollary S.A. A random initialized DeepPipe with encoder layers induces an assumption that two hyperparameter configurations of algorithm should huve more similar performance than hyperparameter configurations from different algorithms.
Proof. Given two hyperparameter configurations X0) (m)) from algorithm, and third hyper- parameter configuration (()) from different algorithm, every random initialized encoder layer from DeepPipe maps the hyperparameters ( (m)) to latent dimensions 200, (m))) that are closer to each other than to 2(x), i.e. the expected distance among the output of the encoder layer will be Eptw> (112* 2" []) < Ep)")) (112* 2*| based on Proposition S.3 Since DeepPipe uses kernel such that <<ær, ar') x(x x²), their similarity will increase, when the distance between two config- urations decreases. Thus, according to the Equation 2 they will have correlated performance.
T META-DATASET SPLITS
We specify the IDs of the task used per split. The ID of the tasks are taken from the original meta-dataset creators.
(i) PMF Meta-Dataset
Meta-training: 4538, 824, 1544, 1082, 1126, 917, 1153, 1063, 722, 1145, 1106, 1454, 4340, 477,
938, 806, 866, 333, 995, 1125, 924, 298, 755, 336, 820, 1471, 1120, 1520, 1569, 829, 958, 997, 472,
1442, 1122, 868, 313, 928, 921, 1446, 1536, 1025, 4534, 480, 723, 835, 1081, 950, 300, 1162, 821,
469, 933, 343, 766, 936, 1568, 785, 31, 164, 761, 1534, 1056, 685, 1459, 230, 867, 828, 161,
742, 1136, 385, 877, 11, 1066, 1532, 1533, 941, 468, 1542, 795, 329, 792, 782, 1131, 796, 4153, 448,
1508, 1065, 1046, 1014, 54, 780, 748, 1150, 793, 1441, 1531, 717, 819, 1151, 287, 1016, 4135, 874,
162. 1148, 1005, 956, 1528, 23, 1516, 446, 1567, 729, 910, 1156, 32, 1041, 1501, 955, 1129, 827,
937, 180, 1038, 973, 36, 44, 1496, 855, 400, 754, 1557, 1413, 758, 817, 1563, 181, 1127, 43,
277, 1141, 715, 725, 884, 790, 880, 853, 155, 223, 1529, 1535, 6, 1009, 744, 1107, 1158, 830, 859,
947, 1475, 813, 734, 976, 227, 1137, 762. 777, 751, 784, 886, 885, 843, 1055, 1486, 1237, 225, 39,
778, 721, 392, 312, 857, 457, 1450, 209, 779, 479, 718, 801, 770, 1049, 391, 12, 730, 759, 1013, 338, 719, 988, 974, 787, 60, 741, 865, 1050, 735, 1079, 1482, 1143, 954, 1020, 1236, 814, 1048, 892, 879,
745, 971, 913, 1152, 694, 1133, 765, 905, 804, 848, 40477, 846, 334, 791, 923, 377, 1530, 889, 1163,
1006, 749, 922, 10, 59, 1541, 310, 461, 1538, 398, 870, 1481, 970, 1036, 1044, 1068, 187, 476, 1157,
40478, 1124, 1045, 845, 62, 915, 1167, 1059, 458, 815, 28, 797, 462, 21, 952, 467, 1505, 375, 882,
1011, 1460, 964, 1104, 275, 732, 189, 478, 1464, 979, 40474, 772, 720, 1022, 823, 811, 463, 61,
1451, 1067, 1165, 184, 716, 962, 978, 916, 1217, 935, 900, 925, 919, 871, 808, 335, 799, 983,
1169, 1004, 837, 1507, 4134. 890, 1062. 1510, 818, 728, 1135, 1147, 1019, 450, 1561, 40476, 816.
1562, 740, 864, 942, 151, 713, 953, 737, 1115, 1123, 1545, 1498, 850, 873, 959, 951, 991, 1132,
1154, 294, 1040, 894, 26, 878, 307, 881, 746, 679, 872, 863, 943, 18, 1537, 767, 794, 1121, 1448,
401, 14, 1026, 833, 875, 1488, 383, 914, 20, 1043, 1116. 292, 847, 1540, 1069, 1155, 1015, 1238, 1149, 1546, 841, 1565, 1556, 1527, 682, 465, 1144, 769, 1517, 756, 834, 912, 807, 904, 16, 1061, 386, 805, 3, 775, 464, 50, 1455, 1021, 1160, 1140, 1489, 1519, 946, 994, 46, 22, 1443, 339, 969, 965, 30, 977, 860, 1500, 1064, 776, 822, 182, 743, 934, 1060, 803, 980, 1539, 346, 788, 1444, 1467, 727,
1509, 903, 832.
Meta- Test: 906, 789, 1159, 1600, 48, 1453, 876, 929, 1012 891, 1164, 726, 459, 812, 909, 927, 774, 278, 279, 1054, 918, 763, 394, 948, 40, 1100, 736, 1503, 1071, 1512, 1483, 53, 869, 285, 773,
28

---

Under review as a conference paper at ICLR 2023
1518, 197, 926, 836, 826, 907, 920, 1080, 1412, 276, 764, 945, 1543, 1472, 996, 908, 896, 851, 397, 783, 1084, 731, 888, 733, 1473, 753, 683, 893, 825, 902, 750, 1078, 8, 1073, 1077, 475, 724, 1513, 384, 388, 887, 714, 771, 1117, 1487, 337, 1447, 862, 838, 949, 800, 331,911.
Meta- Validation: 1075, 747, 901, 1452, 387,,,77 752, 932, 768, 40475, 849, 1564, 1449, 895, 183.
(ii) TensorOBOE Meta-Dataset
Meaining 210, 20, 491, 339, 14, 170, 483, 284, 543, 220, 493, 64, 524, 485, 120, 81, 495, 362, 243, 545, 538. 532, 160, 541, 238, 436, 320, 272, 497, 412, 51, 195, 191, 116, 345, 400, 164, 106, 376, 63, 105, 308, 523, 490, 319, 93, 468, 517. 198, 145, 150, 364.364,
303,
221,
518, 146, 241, 457, 114, 372, 176, 168, 536, 350, 338, 416, 254, 337, 424, 255, 232,
133, 33, 88, 290, 44, 61, 199, 492, 529, 500, 343, 218, 302, 297, 295, 35, 344, 29, 432, 410, 417,
309, 527, 217, 27, 402, 351, 156, 403, 414, 138, 212, 104, 438, 415, 421, 466, 189, 214, 508, 204, 234, 259, 67, 24, 216, 300, 223, 129, 458, 111, 166, 47, 227,, 79, 380, 327,
13, 287, 326, 496, 251, 228, 420, 161, 83, 117, 110, 149, 152, 16, 407, 331, 109, 422, 139,
237, 260, 352, 428, 317, 323, 484. 248, 467, 19, 328, 296, 454, 269, 363, 226, 465, 3, 542, 125, 280, 286, 77, 184, 371, 540, 275, 294, 521, 32, 80, 307, 258, 11, 360, 447, 86, 266, 36, 193, 58, 41, 270, 401, 50, 209, 481, 480, 504, 503, 123, 222, 419, 62, 456, 377, 130, 187, 23, 451, 479,
370, 394, 0, 383, 201, 368, 515, 98, 387, 349, 304, 418, 292, 178, 369, 256, 94, 197, 95, 535, 163. 169, 69, 305, 48, 341, 373, 397, 207, 279, 514, 227, 148, 143, 334, 180, 356, 460, 131, 127,
452, 262, 324, 203, 84, 426, 121, 544, 520, 534, 398, 384, 91, 82, 430, 267, 119, 358, 291, 57, 425, 487, 321, 257, 442, 388. 335, 488, 53, 522, 128, 28, 183, 459, 510, 151, 244, 265, 288, 423, 147, 177, 99, 448, 431, 115, 72, 537, 87, 314, 396, 472, 70, 277, 9, 359, 192
Meta-Test 118, 548, 453, 385, 512, 353, 247, 179, 332, 379, 10, 489, 112, 293, 219, 395, 281, 65, 409, 126,401, 401,526, 342, 346, 413, 167, 38181, 506, 289, 539, 282, 101, 97, 278, 54, 30, 298, 49, 100, 474, 461, 283,
144, 60, 6, 507, 336, 225, 261,38, 329, 365, 445, 429, 513, 188, 469, 124, 154, 59, 473, 498, 546, 528, 263, 194, 55, 171, 236, 206, 158, 196, 34, 408. 18, 501, 250, 533, 52, 74, 173, 92, 167, 382, 181, 208, 354, 249, 450, 5, 141, 525, 200,
135, 531, 122, 22, 68
Meta- Validation 85, 446, 96, 172, 134, 37, 392, 90, 509, 389, 378, 435, 66, 391, 530, 333, 462, 231, 330, 301, 325, 268, 434, 318, 233, 213.549 140, 264, 482, 155, 235, 175, 157, 113, 165, 245, 246, 15, 361, 547, 470, 17, 306. 190, 153, 357, 45, 443, 162, 475, 186, 224, 494, 393, 399, 444, 550, 439, 516, 433, 230, 108, 89, 406, 46, 102, 463, 21, 107,374, 211, 103, 71, 75, 316, 78, 240, 205, 386, 202, 142, 313, 252, 348, 511, 437, 347, 478, 355, 476, 242, 276, 519, 499, 285, 271, 229, 1, 390, 12, 132,
299, 404, 440, 239, 185, 76, 367, 315
(iil) ZAP Meta-Dataset
Meta-Train O-svhn_cropped 1-svhn_croppee
4- 2-svhn_cropped. 3-svhn_cropped,
svhn_ ccon_ero 5-svhn_ cropped, 6-svhn_cropped 7-svhn_croppe _coop_ed 8-svhn_cropped,
svhn cropped, 10-svhn 10-svhn_croppeed 11-svhn cropped, 12-svhn seswhn_cropped. 13-svhn fettt_e_roeed...1111
14-
svhn chop_ce O-cycle_gar gan gaingagnnngaaga apple2orang,, I-cycle gan apple2orange, 2-cycle gan gangggannnggaaeee applee2poe....c
3-cycle_gae Laiiignnnngggg san_apple2orange,
4-cycle_gan laall__annaapee2ranneee
5-cycle --cycle_gan__ sesel__ga__appeeen gaa_aaapee2eoraneeeeeeeee
6-
cycle_ger Llillaan__gg__ee aaa___p222orane,,
7-cycle_gan gai__aaa__a__ aama_pp222oangeee.
8-cycle_gan soocce_gaanapppee gaa_aaapee2eraneeee..
9.
eycle_ge Bal__aannnggggga apple2oaee,,,,, 10-cycle gaaeegann___ 2pplee,, 11-cycle_gam gooll_gan__te lai__aannng_ppgg
12-
cycle_gan_appll 2pg__appl2eoranee, 13-cycle_gan_eee gan_apple2orangee 14-cycle hat___aaaappppepapgeeea
Obcats_vs_dos
rreatsvs_ddope s--cats_vs_dopeee 6-cats rreats_vs_doge vs_gogs, 7- cats vs_ vs_dogs, 8-cats vs ps__o 9-cats ssss_sss_e_e dogs, 10-cats vs dogs. 11 11-cats_ vs dogs, 12-cats vs dogs. 13-cats vs dogs, 14-cats vs dogs, O-stanford dogs, 1-stanford dogs, 2-stanford dogs, 3- stanford_doge 4-stanford _ogs, 5-stanford_ dogs, 6-stanford_dae dogs, 7-stanford_dopp stanford_ dogs, 9-stanford_ _oog,, 10-stanford_ 11-stanford _dogsdddd_os___ooooo_ 12-stanford _dgg, 13- stanford _anss 14-stanford _dggs O-cifar100, 1-cifar100, 2-cifar100, 3-cifar100, 1-c,far
--cat_vs_ddose 2-cats_vs_don
3-cats_vs_doo
8-
1-cifar100, 6-cifar100, 70cifar100, 8-cifar100, 9-cifar100, 10-cifar100. 11-cifar100, 12- cifar100, 13-cifar100, 14-cifar100. 0-coil100, 1-coil100. 2-coil100, 3-coil100, 4-coil100, 5-coil100, 6-coil100, 7-coil100, 8-coil100, 9-coil100, 10-coil100, 11-coil100, 12-coil100. 13- coil100, 14-coil100, O-omniglot, 1-omniglot, 2-omniglot, 3-omniglot, 4-omniglot, 5-omniglot, 6-omniglot, 7-omniglot, 8-omniglot, 9-omniglot, 10-omniglot, 1 11-omniglot, 12-omniglot, 13-omniglot, 14-omniglot, 0-cars196, 1-cars196, 2-cars196, 3-cars196. 4-cars196, 5-cars196, 6-cars196, 7-cars196, 8-cars196, 9-cars196, 10-cars196, 11-cars196 12-cars 196, 13-cars196.
29

---

Under review as a conference paper at ICLR 2023
14-cars196, O-horses or humans, 1-horses humans, 2-horses humans, 3 3-horses or humans, 4-horses or humans, 5-horses or humans, 6-horses or humans, 7-horses or humans, 8- horses or humans, 9-horses or humans, 10-horses humans, 11-horses or humans, 12- horses_or_hunee humans, 13-horses_or caaarsee_or_uumans 14-horses_or coarses_r__humans, 0-tf ffowwes 1-tf 1-tf_flowers 2-1f_ ffowfleers, 3-tf flowws, 4-tf flowers, 5-tf_ fff_wle flo_ffl 7-1f flowerr 8-tf flowers, 10-tf flowers, 11-tf flowers, 12-tf flowers,
flowers, 0- cycle_gan maps, I-cycle Baallle_gnn_nge gan_maps, 2-cycle_ gan maps, 3-cycle_ laiinannnaappa 4-cycle_ _aa__ann___ _aa__apss, cycle_ge _ap__ppsspp,s,,,.._________ 6-cycle_gee maps, 7-cycle gan mams 8-cycle_ge Bacl__ann___appsaaae 9-cycle_gaa Lai__aaa____a__aaaaaa _aps, 10- cycle_gan
fltweflwwer,
gas__mpss_, 11-cycle_s gaa maps, 12-cycle_ gan maps, 13-cycle gan maps, 14-cycle_gat gan maps, scipsrrs B-rock_paper_scissoe --rock_paper_scissonn
O-rock_papp O-rocK_paper_scisste seisssss I beacc_papersssissoee
-rroc__pape__scisonn
Ssrock_paper_siisee scissors, b-rock_paper_scisson 11 [1=rock_paper_sciissn
$-rock_paper_sccisooss 12-rock_paper_seee scissors,
frrnek_paper_sciseee
paisr_s 13-rock paper
Sarrocc_pppppra___ paper_scissors, scissors, 14-rock_pappe
prrock__appe__ttee scissors, O-cassava, I-cassava,
2-cassava, 3-cassava, 4-cassava, 5-cassava. 6-cassava, 7-cassava, 8-cassava, 9-cassava, 10-cassava,
1 1l-cassava, 12-cassava, 13-cassava, 14-cassava, 0- 1-maaaggrii,inmmaaaeea 1-cmaterdb__ce 1-cmaterabbdeeanaagattt
2-ematerdb devanagari, 3-cmaterdb devanagari, 4-cmaterdb beterrddbddddaaariieaeaa 5-cmaterdb -e-emattrrbb_ddvaaaaaatt seinaterdb_devanaggee
cetinaaterd_devanaggeee 8-cmaterdb_ 6-cmaterdb__evvaaagaii,, 9-cmaterdb_ beeererdddddddaaaarrarara 10- ematerdb _everragrdddaaaariii.aaaa 11-cmaterdb _eveanagrrrd 22-mmaterdb_devanaee 13-cmaterdb 13-ematerdbbddevaaagaat devanagari,
14 cmaterdd devanagari,
O-cycle gan vangogh2photo. 1-cycle _aanaaa_____ vangogh2photo, 2-
cycle gan vangogh2photo.
3-cycle gan vangogh2photo.
4-cycle gan vangogg2photto
5-cycle garllegann_eeee vangogh2photo,
6-cycle Bacll_gan___vaeeeeeaa vangogh2photo,
7-cycle_gaad Laclii__gg______e_
10-cycle_gan gaa_ggaaggg22phort
l1-cycle_gan Sare__aaavvaggghhhoooeee
12-cycle_gas socycle_ann_anggghhh22
[
14-cycle_gad gan van__oggggh2phooo
0-cycle_gan gan laiiiagnnnn_____eee
ukiyoe2photo,
l-cycle_ gan men_ukkyoe@photo.
2-cycle_gan garcllegan_ae laiggannnuuia ukiyoe2photo,
3-cycle_gan gan__kkyoeephooo.
uatk______ikkooephoooo. 5-
cycle_gen gaanaannnn pam_ukiyoe2photo,
6-cycle_ged gan sin_uuuyye2hhooo,
7-cycle Bas__gan_uuhee gan_ukiyee2photo,
eycle_ gan lligggnnnnn_gg ukiyoe2photo,
9-cycle_gan satau_uuuupphotoo.
10-cycle gan ukiyoe2photo, 13-cycle_ gan ukiyoe2photo,
11-
cycle gan sim_uiyooe2phoo, cycle_ge gooll_gan_iae ukiyoe2photo, 0-cifar10, 1-cifar10, 2-cifar10, 3-cifar10, 4-cifar10, S-cifar10, 6-cifar10, 7-cifar10, 8-cifar10, 9-cifar10, (0-cifar10, 11-cifar10, 12-cifar10, 13-cifar10, 14-cifar10, 0- ematerdb bangla, 1-ematerdb bangla, 2-cmaterdb bangla, 3-ematerdb bangla, 4-cmaterdb bangla, 5-ematerdbb_ne bangla, 6-cmaterdb__eel crcnalerdbbbangla, 7-cmaterdb bangla, 8-cmaterdb caanaterdbbbaglaa... 9-ematerdb bangla, 10-ematerdh bangla. ematerdb bbagglaaaaaaaaaa O-cycle gan ipspneposoooneffeeeeeeeee
12-cycle gan ukiyoe2photo,
14-
_aaglaa 11-ematerdb _angla, 12-ematerdb _angla, 13-cmaterdb _aanattedbbbaaglaa..__ 14-
1-cycle_gam gaaleegaa___ jds.ppphheedslreee flower, 2-
goce__gan_ihoone2dsrfffee flower, Saryyle_gaa_iphhne2dssffffffeoee flowee wtyycoleganniphonedddsistiitttie
pocl__gan_iphone2ddee flower, horele__aaaiiphoee22sfffffoooee ffowerr htapee__ga_iphhoee2sss_fffoo floweer 8- 1 El-cycle_gan iphone2dslr flower.
sylle_gan_iphome22ssr__. flower, 6-cycle_gan_iphoneeee
flower, 10-cycle_gan Laaeeeegan__iiii gawn__phooeesssffffoeere
12-cycle gan gat__iihoneedslrrrrcreoeeaaaasee fftweeeddll__feerrrrrm flower,
13-
cycle gan jun_iihhon2dslrr flower, 14-cycle_gam gan iphone2dslr flower, O-emnist __ni_mmnnttttttt 1-emnist_mmin Demnist_mnist 3-emnist_mmi _nnist,___ttt_,,iiiiiiii 4-emnist mnist, 5-emnist_ mmist, 6-emnist_ cemmisstmmist, 7-emnist_min 8-emnist_ _nniist 9-emnist_ seemist_mnist, 10-emnist_mnine mnist, 11-emnist_mmnn emnist_mnist,
12-emnist_minn
13-
ceemmist_mmist 0-eurosat_rgb. 1-eurosat_rgh 2-eurosat_rgh, 3-eurosat_rgh 4-eurosat_rgb, 5-eurosat_rgb, 6-eurosat_rgb, 7-eurosat_rgb, 8-eurosat_rgb, 9-eurosat_rgb, 10- eurosat rrns 11-eurosat rgb, 12-eurosat_rghe 13-eurosat_rgee 14-eurosat rgb, O-colorectal histology, I c-oolrrccal histology, 2-colorectal histology 3-colorectal histology, 4-colorectal Aecolorectalhhistologyy. 5-colorectal histology, 6-colorectal histology, 7-colorectal histology. 8-colorectal histology, (D-colorectal_hissoooggy 11-colorectal #icollrectal_histoolggg
9 coolorectal_iisolloge
potllorcctal_histolloo
3-colloectal__hitolloo
Di-colorectal_bistologe O-cmaterdb, O-cmaterdh_telugu. 1-cmaterdb t_tlluuttuuu 2-
ematerdb_teluge 3-cmaterdb_ (tclbbu_euuuuueeeeee 4-cmaterdb_tee setnuterrd_teluuu... 5-ematerdb _ettutrrrrd_teluuuuuuu B-cmaterdb_telugeee 7- ematerdb smaterdb_telugu, @ecmaterdb_telugee 9-cmaterdb_tedt _ttluuuttuuuuuteeeee (0-cmaterdbbtellge 110pu,11-e 11-cmaterdb. 11-omaterdbbttlluuu 12-cmaterdb telugu, 13-cmaterdb telugu, 14-cmaterdb telugu, O-ue _er__e,,d,, ssuu_meredd 2-uc soc_mered, 3-uc_ aac_mereed, suuuueeeeede,eeee 5-uc mereedd 6-uc merced, 7-uc merced, 8-uc ssuc_merced, 9-uc_merced, _ee__ee 11-uc_merced 12-uc_merced 1-kmnist, 2-kmnist, 3-kmnist, 4-kmnist, 5-kmnist, 6-kmnist, 7-kmnist, 8-kmnist, 9-kmnist, 10-kmnist, 11 11-kmnist, 12-kmnist, 13-kmnist, 14-kmnist
_eeeeeddddd 14-uc_merced, 0-kmnist,
Meta-Test 0ccyle_gan_ummme2Winne:: --cycle_gan_sse wttn__mmme2iitteeyoooetoot cycle wwt.uusmmeieeeeeesseeeeeee
cccyle_gan_suumeer222
5-cycle_gar www.ssmmmittteeyssseetaooe
6-
30

---

Under review as a conference paper at ICLR 2023
eycle_ gai_nnnnnnnn
summer2winter_yosemmee
8-
cycle gan_ sentmeerwwitee_yoseeeee
9-cycle_gar www.smmewiieeeyooseeeeee
10-
eycle gan Summer2wimter_yosemme
11-cycle gan summer22inter_yoseee
12-
13-cycle_gand socrycle_aaasuumnrreiiieeeee_e wtt_summreiiieee_oosmiiteee
14-
0-malaria, 1 1-malaria, 2-malaria, 3-malaria, 4-malaria, 5-malaria, 6-malaria, 7-malaria, 8-malaria, 9-malaria, 10-malaria, 11-malaria, 12-malaria, facades,
13-malaria, 14-malaria, O-cycle_gad lasligann__aaacal facades, 1-cycle_gee gan facades, 2-cycle_gas gayclegann_aa lai_ggannnaaaaeeeeeee
3-cycle gan facades, 4-cycle Lasl_aaa__a_aeeee facades, 3-cycle_gad las__ga__aa_eeeee facades 6-cycle gaangaanaaaaaa facafacc,ss 7- D-cycle_gan Ball_aa______eeea facades, 8-cyele _aalle_annn__ gsan_facades 9-cycle gan facades, gaa___aaeaea,,
cycle_gan
10-cycle_gare gan__acadee,, facades, 11-
cycle_gan facades 12-cycle_gad safadesss 13-cycle _aatle_____aaassss___eaa D-emnist_balanced. baaniittballnced, 2-emnist_balancedd 3-emnistt_alaane balanced, 4-emnist_balancee
14-cycle_ge sarcll_gan_facadeee
5-emnist_balanced, 6-emnist_ baapiist_bblaeeedd,eee T-emnist_balanced, ceemnist_balancee 10-emnist balanced,
beemnist_balanced balanieet_bbanedd,,. 12-emnist balanned,, 13-emnist_balance
14-
comnit_bbllanee 0-imagenette, 1-imagenette, 2-imagenette, 3-imagenette, 4-imagenette, 5-imagenette, 6-imagenette, 7-imagenette, 8-imagenette, 9-imagenette, 10-imagenette, 11- imagenette, 12-imagenette, 13-imagenette, 14-imagenette, O-mnist, 1-mnist, 2-mnist, 3-mnist, 4-mnist, 5-mnist, 6-mnist, 7-mnist, 8-mnist, 9-mnist, 10-mnist, 1-mnist, 12-mnist, 13-mnist, 14-mnist, Syyyle_gan_horse2220
l-cycle_gan saaee_aan__heee horse2zebra, seycle_gan_horse2zeerr.
2-cycle_gan _aarss222eeera, 5-cycle gan horse2zebra, 8-cycle gan horse2zebra, 11-cycle_ gan horse2zebra, 14-cycle_gam B--yylle_ga_hhoree2eehle hnoree22eeerr
3-cycle gan horse2zebra, 4-cycle _aan horse2zebra, 7-cycle gan horse2zebra, cycle gan horse2zebra, 10-cycle gan horse2zebra, cycle gan honnhhh2eebaa, tyclegan__horse2rhhre
6-
12-
Meta- Validation 0-emnist_ B 1-emnist L-emnist_byclasss, 2-emnist byclass, _-emnist_byclass,
Deemnist_byclasee
4-emnist byclass, 5-emnist byclass, 6-emnist byclass, 7-emnist kemnist_byclass, demmist_byclass,
_eenaistbbellass,
9-emnist_bychhe byclass, eieemnist_byllsse 11-emnist_byclass, 12-emnist_byclan
bebemnist_byclase 14-
emnist_byclass,
I-imagenett resized__2xx3 -eimaagnet_reszzed___ _3xx32
3-imagenet_resine resized 32x32,
4-imagenet resized 32x32,
5-imagenet resized 32x32,,
6-
resized 32x32,
remiige resized 32x32, 7-imagenet
6-imagenet_revie resized 32x32,
imagenet_resized__
__xx32 32x32,
(D-imageent__reszzedd 33xx2,,
11-imagenet_resizee__222 _3xxx2,
12-
imagenet_resized_333
B--imagenet_resized,
(4-imagenet_resized__3 33x322,
()-
fashion_mnis 1-fashion, mnist, 2-fashion_mnicn 3-fashion mnist, 4-fashion_mman kashion__misst, 5-fashion mnist,
6-fashion_mmine mnnist, 7-fashion
"iisshoon_miis,, mnist, 8-fashion_me _nnist,,_, 9-fashion s_niit,, (O-fashion_mnis)
11-
fashion_mmin mnist, 12-fashion_mmis 13-fashion_miin 14-fashion_mnit
(iv) OpenML Datasets
10101, 12, 146195, 146212, 146606, 146818, 146821, 146822, 146825, 14965, 167119, 167120, 168329, 168330, 168331, 168332, 168335, 168337, 168338, 168868, 168908, 168909, 1689 10, 168911, 168912, 189354, 189355, 189356, 3, 31, 34539, 3917, 3945, 53, 7592, 7593, 9952, 9977, 9981
We checked that there is not overlap between the tasks used for meta-training from the TensorOBOE and the tasks used on OpenML Datasets.
31


Reviews:
[{"review_id": "js8IJ2n6D4", "reviewer": "Reviewer_2otF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel approach to pipeline optimization for machine learning models. The proposed method encodes the hyperparameters of every algorithm into a latent space and then aggregates the hyperparameters of the algorithms from all stages into a single latent representation. Based on this learned latent representation, the algorithm uses Bayesian optimization with deep kernel learning to optimize the pipelines, and also allows meta-learning using existing datasets. Extensive experiments are performed to show the practical efficacy of the propose method.", "summary_of_the_review": "The method proposed in this paper is intuitive and useful, and the experiments are comprehensive and nicely done. My only major concern is the first comment I listed above under \"Weaknesses\" regarding whether some of the algorithmic hyperaprameters have been unfairly fine-tuned for different experiments.", "strength_and_weaknesses": "Strengths:\n- The idea of per-component encoders is particularly interesting since it allows the propose method to be extendable. And this extendability has been empirically verified.\n- The experiments are extensive, and they do clearly show the empirical advantage of the proposed method. The way the experiments section (Section 5) is written makes it particularly easy to understand what questions/hypotheses the experiments are trying to answer.\n- Last paragraph of Section 6, the insight provided in this paragraph is particularly nice.\n- The paper is very well written, the proposed method is well explained.\n\nWeaknesses:\n- Section 5.4, top of page 8: It is worrying that different values of F and numbers of encoder layers are used in different experiments. How are these parameters chosen? Ideally, the algorithm should work well in different experiments without requiring the tuning of these parameters; on the other hand, if these parameters are tuned for every experiment, this will be unrealistic since we usually don't get to do this in practice when faced with a new task. This is my biggest concern, please clarify.\n- The paper claims that the proposed method is able to take into account the interactions among different components, but the paper never made it clear which specific algorithmic design(s) has made this possible. I think this is an important insight and should be explicitly discussed.\n- Last paragraph of Section 6, the insight provided in this paragraph is nice as I said above, but why have you used randomly initialized weights for DeepPipe rather than using the weights after training?\n- Section 5.3, Hypothesis 5: I wonder what's the implication of this hypothesis? In other words, what insights can we draw about your algorithm if this hypothesis is validated (which it is)?\n- Page 8, last paragraph: the presentation of this paragraph can be improved, it's difficult to understand in the current form.\n- [minor] Section 3.2, equation (2): it looks like you are assuming noiseless observations, which is different from what is written in the paragraph above where the observations are noisy. Perhaps the paragraph should be modified to discuss noiseless observations.\n- [minor] Section 4.2, equation (5): should give some intuitions as to how this objective is designed.\n- [minor] Section 5.4: The notations in this section are kind of inconsistent with the previous ones such as those in Section 4.1. Is it true that  $L_i=F(Q_i+M_i)$?\n- [minor] Last line of page 7: I think the left bracket is in the wrong place, I think it should be $F\\sum_i(Q_i+M_i)$", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TbTj9iwmkL", "reviewer": "Reviewer_5R6t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to tackle the pipeline optimization problem that jointly optimizes various hyperparameters in the whole pipeline of machine learning. It proposes per-algorithm encoders and further aggregates the embeddings from all stages of the pipeline to a vector representation. Bayesian optimization is able to efficiently optimize the pipeline performance using the aggregated representation. The paper empirically shows the proposed method’s better performance against baselines. The authors also show the possibility of meta-training the model on existing pipeline evaluation data, and the method’s ability to quickly adapt to different dataset tasks as well as changes in the pipeline structure (e.g., the addition of new algorithms).", "summary_of_the_review": "Overall, the paper conducts many empirical experiments to validate the superior performance of DeepPipe. However, several aspects are lacking. The authors did not fully establish the gap between the literature and this paper. The inability of previous methods in capturing the interaction and relations among pipeline stages is hypothesized, rather than proved or shown in the experiments. Some important baselines are left out in the experiments.", "strength_and_weaknesses": "Strengths:\n1. The paper is nicely structured and clearly written.\n2. The experiments carried out cover rather extensive scenarios.\n3. The authors additionally looked into the inductive bias of the introduced embeddings, which results in some insights for the architecture design.\n\nWeaknesses:\n1. The novelty of the method is limited, in the context of recent literature.\n2. Lack of empirical justification for the proposed method in the general deep learning setting, where pipeline optimization can be of great importance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SO9-rUoyQqN", "reviewer": "Reviewer_KTJe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The selection of the algorithms and their hyperparameters is known as Pipeline Optimization. For most existing Pipeline Optimization, techniques do not explore the deep interaction between pipeline stages/components (e.g. between hyperparameters of the deployed preprocessing algorithm and the hyperparameters of a classifier).  This paper aims to capture the deep interaction between components of a Machine Learning pipeline and introduces DeepPipe, a neural network architecture for embedding pipeline configurations on a latent space. Such deep representations are combined with Gaussian Processes (GP) for tuning pipelines with Bayesian Optimization (BO). In experiments, this article demonstrates that the proposed pipeline representation helps to achieve state-of-the-art results in optimizing pipelines for fine-tuning deep computer vision networks.", "summary_of_the_review": "The authors present some useful feature-processing ideas in AutoML. However, they didn't jump out of the constraint of the given AutoML framework. Moreover, the experimental results are not strong enough. There are only a few baselines, and some of them are too old. As far as I know, they are many BO works that can be compared.", "strength_and_weaknesses": "Questions: \n\n(1) In Equation (1), $p^*$ and $\\lambda(p^*)$  are the parameter to be optimized, and the author has set $p_{\\lambda} = (p,\\lambda(p))$, in section 4, the representation of neural network is $\\phi\\left(p_\\lambda ; \\theta\\right): \\operatorname{dom}\\left(p_\\lambda\\right) \\rightarrow \\mathbb{R}^Z\n$, but Equation (3) does not reflect that $p$ is encoded into MLP, please explain it.\n\n(2) The optimization of $\\theta=\\left\\{\\theta^{\\text {enc }}, \\theta^{\\text {aggr }}\\right\\}$ is confusing. Minimizing the negative log-likelihood of the GP requires a detailed explanation.\n\n(3) The explanation of Equation (5) in Appendix G is too simple and needs a little description.\n\n(4) In figure 1, the 'selector' step can be explained more clearly.\n\nA lot of work has been done and compared with the previous work. From the experimental results, this neural network-based representation method has achieved great success.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject