---
abstract: 'For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them.  However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values.  This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset.  Besides, an in-depth analysis is performed to understand what they take into account for their decisions, providing useful insights.  An extensive analysis of different categories of meta-features, meta-learners, and setups across 156 datasets is performed.  Results show that it is possible to accurately predict when tuning will significantly improve the performance of the induced models.  The proposed system reduces the time spent on optimization processes, without reducing the predictive performance of the induced models (when compared with the ones obtained using tuned hyperparameters).  We also explain the decision-making process of the meta-learners in terms of linear separability-based hypotheses.  Although this analysis is focused on the tuning of Support Vector Machines, it can also be applied to other algorithms, as shown in experiments performed with decision trees.' 
address:
- 'Institute of Mathematics and Computer Sciences, University of São Paulo, São Carlos - SP, Brazil'
- ' Eindhoven University of Technology, Eindhoven, Netherlands'
- 'Universidade Estadual Paulista, Campus de Itapeva, São Paulo, Brazil'
- 'Federal Technology University, Campus of Apucarana - PR, Brazil'
author:
- 'Rafael G. Mantovani'
- 'André L. D. Rossi'
- Edesio Alcobaça
- |
    \
    Joaquin Vanschoren
- 'Andr[é]{} C. P. L. F. [de Carvalho]{}'
title: 'A meta-learning recommender system for hyperparameter tuning: predicting when tuning improves SVM classifiers'
---

Meta-learning ,Recommender system ,Tuning recommendation ,Hyperparameter tuning ,Support vector machines

Introduction {#sec:intro}
============

Many algorithms, among them  [@vapnik:1995], have been successfully used in a wide variety of problems.  are kernel-based algorithms that perform non-linear classification using a hyperspace transformation, i.e., they map data inputs into a high-dimensional feature space where the problem is possibly linearly separable.  As most algorithms, are sensitive to their values, which directly affect their predictive performance and depend on the data under analysis.  The predictive performance of is mostly affected by the values of four : the kernel function ($k$), its width ($\gamma$) or polynomial degree ($d$), and the regularized constant ($C$).  Hence, finding suitable is a frequently studied problem [@Horn:2016; @Padierna:2017].  tuning is commonly modeled as a black-box optimization problem whose objective function is associated with the predictive performance of the SVM induced model.  Many optimization techniques have been proposed in the literature for this problem, varying from a simple to the state of the art technique [@Snoek:2012].  In [@Bergstra:2012], Bergstra & Bengio showed theoretically and empirically that is a better alternative than and is able to find good settings when performing tuning.  Mantovani et.  al.  [@Mantovani:2015a] also compared with meta-heuristics to tune .  A large amount of empirical experiments showed that generates models with predictive performance as effective as those obtained by meta-heuristics. 

However, regardless the optimization technique, hyperparameter tuning usually has a high computational cost, particularly for large datasets, with no guarantee that a model with high predictive performance will be obtained.  During the tuning, a large number of settings usually need to be assessed before a “good” solution is found, requiring the induction of several models, multiplying the learning cost by the number of settings evaluated.  Besides, several aspects, such as the complexity of a dataset, can influence the tuning cost. 

When computational resources are limited, a commonly adopted alternative is to use the default values suggested by tools.  Previous works have pointed out that for some datasets, tuning of is not necessary [@Ridd:2014].  Using default values largely reduce the overall computational cost, but, depending on the dataset, can result in models whose predictive performance is significantly worse than models produced by using tuning.  The ideal situation would be to recommend the best alternative, default or tuned values for each new dataset. 

In this paper, we propose a recommender system to predict, when applying to a new dataset, whether it is better to perform tuning or it is sufficient to use default values.  This system, based on  [@Brazdil:2009], is able to reduce the overall cost of tuning without significant loss in predictive performance.  Another important novelty in this study is a descriptive analysis of how the recommendation occurs.  Although the recommender system is proposed for the tuning of , it can also be used for other algorithms.  To illustrate this aspect, we present an example where the recommender system is used for tuning of a induction algorithm. 

The proposed recommender system can also be categorized as an solution [@Feurer:2015b], since it aims to relieve the user from the repetitive and time-consuming tuning task, automating the process through .  The area is relatively new, and there still many questions to be addressed.  This fact, and the emerging attention it has attracted from important research groups [@Feurer:2015b; @Kotthoff:2016] and large companies [^1] [^2], highlights the importance of new studies in this area.  An essential aspect for the success of systems is to provide an automatic and robust tuning system, which also emphasizes the relevance of the problem investigated in this paper. 

In summary, the main contributions of this study are:

-   the development of a modular and extensible framework to predict when default values provide accurate models, saving computational time that would be wasted on optimization with no significant improvement;

-   a comparison of the effectiveness of different sets of meta-features and preprocessing methods for meta-learning, not previously investigated;

-   reproducibility of the experiments and analyses: all the code and experimental results are available to reproduce experiments, analyses and allow further investigations[^3]. 

It is important to mention that we considered the proposed framework for predictive tasks, in particular, supervised classification tasks using .  However, the issues investigated in this paper can be easily extended to other tasks (such as regression) and other algorithms [^4]. 

This paper is structured as follows: Section \[sec:meta-learning\] presents the basic concepts used in our approach.  Section \[sec:related\] defines the tuning problem and presents a concise survey of prior work combining with in some way.  The complete experimental methodology covered to obtain the results is presented in Section \[sec:methods\].  Results are discussed in Section \[sec:results\] while final considerations and conclusions are presented in Section \[sec:conclusions\]. 

Background on Meta-Learning {#sec:meta-learning}
===========================

Several algorithms have been proposed for prediction tasks.  However, since each algorithm has its inductive bias, some of them can be more appropriate for a particular data set.  When applying a algorithm to a dataset, a higher predictive performance can be obtained if an algorithm whose bias is more adequate to the dataset is used.  The recommendation of the most adequate algorithm for a new dataset is investigated in an research area known as  [@Brazdil:2009]. 

has been largely used for algorithm selection [@Ali:2006], and for ranking [@Soares:2006] and prediction [@Reif:2014] of predictive performance of algorithms.  It investigates how to learn from previous experiments.  According to Brazdil et.  al.  [@Brazdil:2009], meta-learning can be used to improve the learning mechanism itself after each training process.  In , the process of using a learning algorithm to induce a model for a data set is called base-learning.  At the meta-level, likely useful information extracted from this process (meta-features) are used to induce a meta-model.  This meta-model can recommend the most promising learning algorithm, a set of the $N$ best learning algorithms or a ranking of learning algorithms according to their estimate predictive performance for a new dataset.  The knowledge extracted during this process is called meta-knowledge.  The meta-features extracted from each dataset is a critical aspect.  They must be sufficient to describe the main aspects necessary to distinguish the predictive performance obtained by different learning algorithms when applied to this dataset.  As a result, it should allow the induction of a meta-model with good predictive performance.  According to [@Vilalta:2004] three different sets of measures can be applied to extract meta-features:

-   *Simple*, *Statistical* and *Information-theoretic* meta-features  [@Brazdil:1994B]: these consist of simple measures about the input dataset, such as the number of attributes, examples and classes, skewness, kurtosis and entropy.  They are the most explored subset of meta-features in literature [@Feurer:2015b; @Gomes:2012; @Miranda:2012; @Reif:2012; @Reif:2014; @Soares:2004];

-   *Model-based* meta-features [@Bensusan:2000]: these are a set of properties of a model induced by a algorithm for the dataset at the hand.  For instance, if a decision tree induction algorithm is applied to the dataset, statistics about nodes, leaves and branches can be used to describe the dataset.  They have also been used frequently in literature [@Reif:2012; @Reif:2014];

-   *Landmarking* [@Pfahringer:2000]: the predictive performance obtained by models induced by simple learning algorithms, called landmarkers, are used to characterize a dataset.  These measures were explored in studies such as [@Feurer:2015b; @Reif:2014]. 

Recently, new sets of measures have been proposed and explored in literature:

-   *Data complexity* [@Ho:2002]: this is a set of measures which analyze the complexity of a problem considering the overlap in the attributes values, the separability of the classes, and geometry/topological properties.  They have been explored in [@Garcia:2016]; and

-   *Complex networks* [@Morais:2013]: measures based on complex network properties are extracted from a network built with the data instances.  These measures can only be extracted from numerical data.  Thus, preprocessing procedures are required for their extraction.  They were explored in [@Garcia:2016]. 

Meta-learning for Hyperparameter tuning {#sec:related}
=======================================

As previously mentioned, there is a large number of studies investigating the use of to automate one or more steps in the application of algorithms for data analysis tasks.  These studies can be roughly grouped into the following approaches, according to what does:

-   it recommends settings;

-   it predicts training runtime;

-   it recommends initial values for optimization;

-   it estimates predictive performance for an setting;

-   it predicts tuning improvement/necessity. 

Table \[tab:svm\_related\] summarizes a comprehensive list of studies that either embedded or used to cope with the tuning problem.  Next, these works are described in more detail. 

Recommendation of HP settings
-----------------------------

The first approach considered settings as independent algorithm configurations and predicted the best setting based on characteristics of the dataset under analysis.  In this approach, the settings are predicted without actually evaluating the model on the new dataset [@Soares:2004].  In Soares et.  al.  [@Soares:2004] and Soares & Brazdil [@Soares:2006], the authors predicted the width ($\gamma$) of the Gaussian kernel for regression problems.  A finite set of $\gamma$ values was investigated for $42$ regression problems and the predictive performance was assessed using $10$-fold and the evaluation measure.  The recommendation of $\gamma$ values for new datasets used a meta-learner. 

Ali & Smith-Miles [@Ali:2006] presented a similar study but selected one among five different kernel functions for $112$ classification datasets.  They assessed model predictive performance for different settings using $10$- procedure and the simple measure.  Miranda & Prud[ê]{}ncio [@Miranda:2013] proposed another approach, called  [@Leite:2012], to select the $\gamma$ and the soft margin ($C$).  Experiments performed on $60$ classification datasets assessed the settings using a single $10$- and the measure. 

Lorena et.  al.  [@Lorena:2018] proposed a set of complexity meta-features for regression problems.  One of the case studies evaluated was the tuning problem.  The authors generated a finite grid of $\gamma$, C and $\epsilon$ (margin of tolerance for regression ) values, assessing them with a single 10-fold and measure, considering $39$ regression problems.  The recommendation of for new unseen datasets was performed by a distance-based meta-learner. 

Prediction of Training Runtime
------------------------------

Other works investigated the use of to estimate the training time of classification algorithms when induced by different settings.  In Reif et.  al.  [@Reif:2011], the authors predicted the training time for several classifiers, including .  They defined a discrete grid of $\gamma \times C$ settings, assessing these settings on $123$ classification datasets considering the and the performance measures.  In Priya et.  al [@Priya:2012], the authors conducted a similar study but used a to optimize parameters and perform meta-feature selection of six meta-learners.  Experiments were carried out over $78$ classification datasets assessing settings using a $5$-fold and the evaluation measure. 

Recommendation of initial values for HP optimization
----------------------------------------------------

has also been used to speed up the optimization of values for classification algorithms [@Feurer:2015b; @Gomes:2012; @Miranda:2012; @Reif:2012].  In Gomes et.  al.  [@Gomes:2012] is used to recommend settings as initial search values by the and optimization techniques.  Experiments were conducted in $40$ regression datasets adjusting the $C$ and $\gamma$ to reduce the value.  A meta-learner was used to recommend the initial search values. 

Reif et.  al.  [@Reif:2012] and Miranda et.  al.  [@Miranda:2012] investigated, respectively, the use of and different versions of for the same task.  In Miranda et.al.  [@Miranda:2014], the authors used multi-objective optimization to optimize the to increase predictive the performance and the number of support vectors.  These studies used simple accuracy measure and $10$-fold to optimize $\gamma \times C$ values. 

The same approach is explored in a tool to automate the use of algorithms, the `Auto-skLearn` [@Feurer:2015b].  In this tool, is used to recommend settings for the initial population of the optimization technique.  The authors explored all the available in $140$ classification datasets.  It is the first and perhaps the only work that uses nested- to assess settings.  Each setting was assessed in terms of the simple measure. 

Estimation of predictive performance for an HP setting
------------------------------------------------------

A more recent approach uses to estimate algorithms’ performance considering their .  In Reif et.  al.  [@Reif:2014], the authors evaluated different algorithms in $54$ datasets, including , and used the performance predictions to develop a system for automatic algorithm selection. 

Wistuba et.  al.  [@Wistuba:2018] adapted the acquisition function of surrogate models by one optimized meta-model.  They evaluated several configurations in a holdout fashion procedure over $105$ datasets and used the meta-knowledge to predict the performance of new settings for new datasets.  The authors also proposed a new that extended the original proposal by predicting the predictive performance of settings for surrogate models. 

Eggensperger et.  al.  [@Eggensperger:2018] proposed a benchmarking approach of “surrogate scenarios”, which extracts meta-knowledge from optimization and algorithm configuration problems, and approximates the performance surface by regression models.  One of the $11$ meta-datasets explored in the experimental setup has a set of settings assessed for the MNIST dataset.  These settings were obtained executing a simple method and three optimizers:  [@Hutter:2009],  [@Lopez:2016], and . 

[lclccccccclcc]{}

& & & & & & & & &\
& & & & & & & & &\
& & & **$k$** & **$\gamma$** & **$C$** & **$d$**\

------------------------------------------------------------------------

Soares et al.  [@Soares:2004] & 2004 & Recommends settings & & $\bullet$ & & & & & 42 & UCI, METAL & 10-CV & NMSE\

------------------------------------------------------------------------

Soares & Brazdil [@Soares:2006] & 2006 & Recommends settings & & $\bullet$ & & & & & 42 & UCI, METAL & 10-CV & NMSE\

------------------------------------------------------------------------

Ali & Smith-Miles [@Ali:2006] & 2006 & Recommends settings & $\bullet$ & & & & & C5.0 & 112 & UCI, KDC & 10-CV & Acc\

------------------------------------------------------------------------

Miranda & & & & & & & & & & & & &\
 Prud[ê]{}ncio [@Miranda:2013]\

------------------------------------------------------------------------

Lorena & et al.  [@Lorena:2018] & 2018 & Recommends settings & & $\bullet$ & $\bullet$ & & & & 39 & UCI & 10-CV &\

------------------------------------------------------------------------

& & & & & & & & & & & & PMCC\
& & & & & & & & & & & & NAE\

------------------------------------------------------------------------

& & & & & & & & J48, & & & &\
& & & & & & & & Bagging\
& & & & & & & & ,NB\
& & & & & & & & Jrip\

------------------------------------------------------------------------

& & Recommends initial values & & & & & & & & & &\
& & for optimization & & & & & ,\

------------------------------------------------------------------------

& & Recommends initial values & & & & & & & & & &\
& & for optimization\

------------------------------------------------------------------------

& & Recommends initial values & & & & & & & & & &\
& & for optimization\

------------------------------------------------------------------------

& & Recommends initial values & & & & & & & & & &\
& & for optimization\

------------------------------------------------------------------------

& & Recommends initial values for & & & & & & & & & &\
& & for optimization\

------------------------------------------------------------------------

& & Estimates predictive performance & & & & & & & & & & RMSE\
& & for a setting & & & & & & & & & & PMCC\

------------------------------------------------------------------------

& & Estimates predictive performance & & & & & & & & & & -\
& & for a setting\

------------------------------------------------------------------------

& & Estimates predictive performance & & & & & , & & & & & RMSE\
& & for a setting & & & & & , & & & & &\

------------------------------------------------------------------------

Ridd & & & & & & & & & J48, RF & & & &\
Giraud-Carrier [@Ridd:2014] & & & & & & & &\

------------------------------------------------------------------------

& & & & & & & , , & J48, SVM, LR & & & &\
& & & & & & & , & NB, , RF\

------------------------------------------------------------------------

Sanders & & & & & & & & & & & & &\
Giraud-Carrier [@Sanders:2017] & & & & & & & &\

Prediction of HP tuning improvement/necessity {#subsec:tuning_necessity}
---------------------------------------------

Although the studies mentioned in this section are the most related to our current work regarding the proposed modeling, they have different goals.  While Ridd & Giraud-Carrier  [@Ridd:2014] and Sanders & Giraud-Carrier [@Sanders:2017] are concerned with predicting tuning improvement, Mantovani et al.  [@Mantovani:2015b] and the present study aimed to predict when tuning is necessary. 

Ridd & Giraud-Carrier [@Ridd:2014] investigated a  problem.  They carried out experiments using technique to search the hyperspace of this  problem in $326$ binary classification datasets.  Their -based method predicts whether tuning would lead to a considerable increase in accuracy considering a pool of algorithms, including .Even though this is one of the first studies in this direction, we could point out some drawbacks:

-   the proposed method does not identify which algorithm and correspondent values the user should run to achieve an improved performance;

-   there is no guarantee that training and testing data are not mutually exclusive;

-   the rule to label the meta-examples is defined empirically, based on thresholds of the difference of the accuracy between default and tuned HP values;

-   all the datasets are binary classification problems; and

-   it is not possible to reproduce the experiments, specially base-level tuning since most of the details are not explained, and the code is not available. 

Sanders & Giraud-Carrier [@Sanders:2017] used a technique for tuning of three different algorithms, including .  Their experimental results with $229$   classification datasets showed that tuning almost always yielded significant improvements compared to default values.  Thus, they focused on the regression task of predicting how much improvement can be expected by tuning compared to default values.  They also addressed this task using .  However, their study presents some limitations, such as:

-   the optimization process of hyperparameters were computationally costly and did not finish for most of the datasets;

-   the meta-learner was not able to predict hyperparameter tuning improvements for in those datasets whose tuning process finished;

-   there is no guarantee that the generated meta-examples are different from each other (intersection between training and test data), since OpenML stores different versions of the same dataset.  This could lead to biased results; and

-   experiments are not reproducible since most of the details are not explained, and the code is not available. 

Mantovani et al.  [@Mantovani:2015b] proposed a  recommender system to predict when   tuning is necessary, i.e., when tuning is likely to improve the generalization power of the models.  The meta-dataset was created by extracting characteristics based on simple and data complexity measures from $143$ classification datasets.  In the base-level, different meta-heuristics (PSO, GA and EDA) were used to tune the SVM using a nested- resampling strategy.  An ensemble of meta-models achieved the best predictive performance assessed by the F-Score using simple meta-features.  Besides these promising results, this study presents some shortcomings, such as:

-   the best predictive performance at the meta-level is moderately low;

-   when the method recommends tuning, the meta-heuristic which would lead to the best performance is not recommended;

-   only two default settings were investigated.  In general, users try more than two settings before tuning;

-   there is no evidence that this method and the results can be generalized to other algorithms. 

The main differences between the proposed approach and the most related work are shown in Table \[tab:most\_related\_work\].  It is important to note that although these are the most similar studies we have found in the literature, they addressed different problems.  Furthermore, the meta-datasets generated by each study were also different, since they were generated using different datasets, target algorithms, meta-features, and labeling rules.  Because of these particularities, the straightforward comparison of these studies is unfeasible.  The only free choice we could explore is the same meta-features adopted by them.  In fact, Ridd & Giraud-Carrier [@Ridd:2014] and Sanders & Giraud-Carrier [@Sanders:2017] used a total of $68$ meta-features which are included in our experimental setup (Section \[subsec:metafeatures\]). 

Based on the literature, we realized that there is room for improvement in terms of predicting tuning necessity for algorithms and to better understand this meta-learning process.  Our present work attempts to fill this gap yielding meta-models with high predictive performance and reasons why their decisions were made.  To do this, we have comprehensive and systematically evaluated different categories of meta-features and preprocessing tasks, such as meta-feature selection and data balancing, and different default values. 

[llclccl]{}

& & **Tuning setup** & **Labeling** & **Target** & **Meta**\
& **prediction** & **Task** & **(base-level)** & **rule** & **algorithm** & **features**\

------------------------------------------------------------------------

& & & & & & Simple\
Ridd & & & & & Accuracy & CASH & Statistical\
Giraud-Carrier [@Ridd:2014] & & & & threshold & (20 algs.) &  Landmarkers\
& & & & & & Model-based\

------------------------------------------------------------------------

& & & Nested-CVs & & &\
Mantovani & tuning & & holdout (inner) & Confidence & & Simple\
 et al.  [@Mantovani:2015b] & necessity & & 10-CV (outer) & interval & & Data complexity\
& & & BAC (fitness)\

------------------------------------------------------------------------

& & & & & CART & Simple\
Sanders & & & & 10-CV (single) & Confidence & MLP & Statistical\
Giraud-Carrier [@Sanders:2017] & & & AUC (fitness) & interval & SVM & Landmarkers\
& & & & & & Model-based\

------------------------------------------------------------------------

& & & Nested-CVs & & &\
& tuning & & 3-fold (inner) & Wilcoxon & SVM & Many\
& necessity & & 10-fold (outer) & test & J48 & (see Section \[subsec:metafeatures\])\
& & & BAC (fitness)\

Summary of Literature Overview
------------------------------

The literature review carried out by the authors found a large increase in the use of for tasks related to tuning.  The authors found $18$ related works, but only three of them investigated specifically when tuning is necessary or its improvement (see Section \[subsec:tuning\_necessity\]).  Overall, the following aspects were observed:

-   fourteen of the studies created the meta-knowledge using to tune the $\gamma \times C$ ;

-   most of the studies also evaluated the resultant models with a single procedure and the simple evaluation measure;

-   half of the studies used in most $100$ datasets.  In [@Ridd:2014], the authors used more than $300$ datasets, but all of them for binary classification;

-   all investigated a small number of categories to generate meta-features;

-   nine of the studies used only as meta-learner;

-   three of the studies applied meta-feature selection techniques to the meta-features;

-   two of the studies provided the complete resources necessary for the reproducibility of experiments;

-   None of the studies combined all these six previous issues. 

In order to provide new insights in the investigation of how the use of in the tuning process can affect its predictive performance, this paper extends previous works by exploring:

-   Meta-features produced by measures from different categories;

-   Use of different learning algorithms as meta-learners;

-   Adoption of a reproducible and rigorous experimental methodology at base and meta-learning levels; and

-   Assessment of the use of meta-feature selection techniques to evaluate and select meta-features. 

One of the main contributions of this paper is the analysis of the meta-model predictions to identify when it is better to use default or tuned values for the , and which meta-features have a major role in this identification. 

Experimental methodology {#sec:methods}
========================

In this paper, experiments were carried out using ideas to predict whether hyperparameter tuning can significantly improve induced models, when compared with performance provided by their default hyperparameter values[^5].  The framework treats the recommendation problem as a binary classification task and is formally defined as follows:

Let $\mathcal{D}$ be the dataset collection.  Each dataset $d_{j} \in \mathcal{D}$ is described by a vector $mf(d_j) = (mf_{j,1},...,mf_{j,K})$ of $K$ meta-features, with $mf_{j,k} \in \mathcal{M}$, the set of all known meta-features.  Additionally, let $\Omega$ be a statistical labeling rule based on the prior evaluations from tuned and default hyperparameter settings ($\mathcal{P}$).  Given a significance level $\alpha$, $\Omega$ maps prior performances to a binary classification task: $\Omega: P \times \alpha \rightarrow C\:|\: C = \{tune, not\_tune\}$. Thus, we can train a *meta-learner* $\mathcal{L}$ to predict whether optimization will lead to significant improvement on new datasets $d_{i} \notin \mathcal{D}$, i.e.: $$\mathcal{L}: \mathcal{M} \times \Omega \rightarrow C$$

Figure \[fig:mtl-framework\] shows graphically the general framework, linking two-level learning steps: the *base level*, where the hyperparameter tuning process is performed for different datasets ($\mathcal{D}$); and the *meta level*, where the meta-features ($\mathcal{M}$) from these datasets are extracted, the meta-examples are labeled according to tuning experiments ($\Omega$, $\mathcal{P}$) and the recommendation to a new unseen dataset occurs ($d_i \notin \mathcal{D}$).  Further subsections will describe in detail each one of its components. 

![ Meta-learning system to predict whether hyperparameter tuning is required (Adapted from [@Mantovani:2015b]).  At the figure, ”mf” means meta-feature.[]{data-label="fig:mtl-framework"}](01_meta-learning-INS_paper.pdf){width="\textwidth"}

OpenML classification datasets {#subsec:datasets}
------------------------------

The experiments used datasets from  [@Vanschoren:2014], a free scientific platform for standardization of experiments and sharing empirical results.  supports reproducibility since any researcher can have access and use the same data for benchmark purposes.  A total of $156$ binary and multiclass classification datasets ($\mathcal{D}$) from different application domains were selected for the experiments (Item 01 in Figure \[fig:mtl-framework\]).  From all the available and active datasets, those meeting the following criteria were selected:

-   number of features does not exceed $1,500$;

-   number of instances between $100$ and $50,000$;

-   must not be a reduced, modified or binarized version of the original classification problem[^6];

-   must not be an adaptation of a regression dataset;

-   all the classes must have at least $10$ examples, enabling the use of stratified $10$-fold resampling. 

These criteria are meant to ensure a proper evaluation (a-b), e.g. datasets should not be so small or so large that they cause memory problems; they should not be too similar (c-d) (to avoid data leakage in our evaluation); and allow the use of 10-fold stratified resampling, given the high probability of dealing with imbalanced datasets (e).  We also excluded datasets already used in our related work on defining optimized defaults, resulting in 156 datasets to be used in our meta-dataset.  All datasets meeting these criteria and their main characteristics are presented on the study page at OpenML[^7]. 

In order to be suitable for , datasets were preprocessed: any constant or identifier attributes were removed; the logical attributes were converted into values $\in \{0,1\}$; missing values were imputed by the median for numerical attributes, and a new category for categorical ones; all categorical attributes were converted into the 1-N encoding; all attributes were normalized with $\mu = 0$ and $\sigma=1$. The `OpenML` [@OpenML:2017][^8] package was used to obtain and select datasets from the `` website, while functions from the `mlr` [@mlr:2016][^9] package were used to preprocess them. 

SVM hyperparameter space {#subsec:hp_space}
------------------------

The hyperparameter space used in the experiments is presented in Table \[tab:hp-space\].  For each hyperparameter, the table shows its symbol, name, type, range/options, scale transformation applied, default values provided by LibSVM [@Chang:2011] and whether it was tuned.  Here, only the kernel is considered since it achieves good performances in general, may handle nonlinear decision boundaries, and has less numerical difficulties than other kernel functions (e.g., the values of the polynomial kernel may be infinite) [@Hsu:2007].  For $C$ and $\gamma$, the selected range covers the hyperspace investigated in [@Ridd:2014].  LibSVM default values are $C = 1$, and $\gamma = 1/N$, where $N$ is the number of features of the dataset under analysis[^10]. 

its symbol, name, type, range/options, scale transformation applied, default values and whether it was tuned. 

\[tab:hp-space\]

[cllcccc]{} **Symbol** & **Hyperparameter** & **Type** & **Range/Options** & **Scale** & **Default** & **Tuned**\

------------------------------------------------------------------------

k & kernel & categorical &{RBF} & - & RBF & **x**\

------------------------------------------------------------------------

C & cost & real & $[2^{-15}, 2^{15}]$ & log & 1 &\

------------------------------------------------------------------------

$\gamma$ & width of the kernel & real & $[2^{-15}, 2^{15}]$ & log & ${{}^{1}\!/_{N}}$ &\

Hyperparameter tuning process {#subsec:tuning}
-----------------------------

The hyperparameter tuning process is depicted in Figure \[fig:mtl-framework\] (Item 2).  Based on the defined hyperspace, hyperparameters were adjusted through a technique for all datasets selected.  The tuning process was carried out using nested resamplings [@Krstajic:2014], an “*unbiased performance evaluation methodology*” that correctly accounts for any overfitting that may occur in the model selection (considering the hyperparameter tuning).  In fact, most of the important/current state of the art studies, including the `Auto-WEKA`[^11] [@Kotthoff:2016; @Thornton:2013] and `Auto-skLearn`[^12] [@Feurer:2015b] tools, have been using the nested methodology for hyperparameter selection and assessment.  Thus, nested-s were also adopted in this current study.  The number of outer folds was defined as $M=10$ such as in [@Krstajic:2014].  Due to runtime constraints, the number of inner folds was set to $N=3$.

A budget with a maximum of $300$ evaluations per (inner) fold was considered.  A comparative experiment using different budget sizes for was presented in [@Mantovani:2015a].  Results suggested that only a few iterations are required to reach good solutions in the optima hyperspace region.  Indeed, in most of the cases, tuning has reached good performance values after $250$-$300$ steps.  Among techniques used by the authors, the was able to find near-optimum hyperparameter settings like the most complex tuning techniques did.  Overall, they did not show statistical differences regarding performance and presented a runtime lower than population-based techniques[^13]. 

Hence, the tuning setup detailed in Table \[tab:hp\_setup\] generates a total of $90,000 = 10$ (outer folds) $\times\:3$ (inner folds) $\times\:300$ (budget) $\times\:10$ (seeds) settings during the search process for a single dataset.  Tuning jobs were parallelized in a cluster facility provided by our university[^14] and took four months to be completed. 

[@lll]{} **Element** & **Method** & **R package**\

------------------------------------------------------------------------

Tuning techniques & Random Search & `mlr`\

------------------------------------------------------------------------

Base Algorithm & Support Vector Machines & `e1071`\

------------------------------------------------------------------------

Outer resampling & 10-fold cross-validation & `mlr`\

------------------------------------------------------------------------

Inner resampling & 3-fold cross-validation & `mlr`\

------------------------------------------------------------------------

Optimized measure & $\{$Balanced per class accuracy$\}$ & `mlr`\

------------------------------------------------------------------------

& $\{$Balanced per class accuracy, &\
& Optimization paths $\}$ &\

------------------------------------------------------------------------

Budget & 300 iterations\

------------------------------------------------------------------------

& $10$ times with different seeds & -\
& seeds = $\{1, \ldots, 10\}$ & -\

------------------------------------------------------------------------

& LibSVM defaults & `e1071`\
& optimized defaults\

\[tab:hp\_setup\]

Meta-features {#subsec:metafeatures}
-------------

The meta-datasets used in the experiments were generated out of ‘*meta-features*’ ($\mathcal{M}$) describing each dataset (Figure \[fig:mtl-framework\] - Item 3).  These meta-features were extracted by applying a set of measures $mf_{i}$ to the original datasets which obtain likely relevant characteristics from these datasets.  A tool was developed to extract the meta-features and can be found on `GitHub`[^15], as presented in Table \[tab:repo\].  We extracted a set of $80$ meta-features from different categories, as described in Section \[sec:meta-learning\].  The set includes all the meta-features explored by the studies described in Subsection \[subsec:tuning\_necessity\].  The exact number of meta-features used from each category can be seen in Table \[tab:mf-categories\].  A complete description of them may be found in Tables \[tab:mfeats\_1\] and \[tab:mfeats\_2\] (\[app:features\]). 

[llrl]{}

**Acronym** & **Category** & **\#N** & **Description**\

------------------------------------------------------------------------

SM & Simple & 17 & Simple measures\

------------------------------------------------------------------------

ST & Statistical & 7 & Statistical measures\

------------------------------------------------------------------------

IN & Information-theoretic & 8 & Information theory measures\

------------------------------------------------------------------------

MB & Model-based (trees) & 17 & Features extracted from decision tree models\

------------------------------------------------------------------------

LM & Landmarking & 8 & The performance of some ML algorithms\

------------------------------------------------------------------------

DC & Data Complexity & 14 & Measures that analyze the complexity of a problem\

------------------------------------------------------------------------

CN & Complex Networks & 9 & Measures based on complex networks\

------------------------------------------------------------------------

& **Total** & 80\

Meta-targets {#subsec:metadatasets}
------------

The last meta-feature is the meta-target, whose value indicates whether the tuning significantly improved the predictive performance of the model, compared with the use of default values.  Since the tuning experiments contain several and diverse datasets, many of them may be imbalanced.  Hence, the  measure [@Brodersen:2010] was used as the fitness value during tuning, as well as for the final model assessment at the base-level learning[^16]. 

The so-called “*meta-label rule*” $\Omega$ (Item 4 - Figure \[fig:mtl-framework\]) applies the Wilcoxon paired-test to compare the solutions achieved by the technique ($\mathcal{P}_{tun}$) and the default settings ($\mathcal{P}_{def}$).  Given a dataset $d_i \in \mathcal{D}$ and a significance level ($\alpha$), if the tuned solutions were significantly better than those provided by defaults, its corresponding meta-example is labeled as ‘`Tuning`’ ($C_{tun}$); otherwise, it receives the ‘`Default`’ label ($C_{def}$). 

When performing the Wilcoxon test, three different values of $\alpha=\{0.10,\:0.05,\:0.01\}$ were considered, resulting in three meta-datasets with different class distributions (Item 5 - Figure \[fig:mtl-framework\]).  The different significance levels ($\alpha$) influence how strict the recommending system is when evaluating if tuning improved models’ performance compared to the use of default values.  The smaller the significance stricter it is, i.e., there must be greater confidence than the tuned hyper-parameter values obtained by improving the performance of the induced models.  It may also imply in different labels for the same meta-example when evaluating different $\alpha$ values.  The initial experimental designs only compared LibSVM suggested default values with the tuned solutions.  The resulting meta-datasets presented a high imbalance rate, prevailing the “`Tuning`” class.  It was difficult to induce a meta-model with high predictive performance using this highly imbalanced data.  An alternative to deal with this problem was to consider the optimized default values proposed in [@Mantovani:2015c].  The optimized default values were obtained optimizing a common set of values, able to induce models with high predictive performance, for a group of datasets. 

![ Average balanced per class accuracies comparing LibSVM default (libsvm.defaults), Multiple optimized default hyperparameter settings (multiple.defaults) and Random Search tuning technique (random.search) when defining the meta-target of each meta-example.[]{data-label="fig:multiple_defaults"}](02_Comparison_MultipleDefaults_Random.png){width="\textwidth"}

Figure \[fig:multiple\_defaults\] illustrates the benefits of using multiple default settings: LibSVM and optimized default values.  In this figure, the x-axis identifies datasets by their OpenML ids, listing them decreasingly by the balanced per class accuracy performances (y-axis) obtained using LibSVM defaults hyperparameter values.  This figure shows three different curves:

-   `libsvm.defaults`: a black dotted line representing the averaged performance values obtained using LibSVM default hyperparameter values.  It represents the choice of a user using LibSVM defaults;

-   `random.search`: a green line representing the averaged performance values obtained using the technique for tuning.  It represents the choice of always tuning hyperparameters; and

-   `multiple.defaults`: a red line representing the best choice considering the LibSVM and optimized defaults hyperparameter values.  It represents our approach, exploring multiple default values. 

By looking at the difference between the black and green lines, it is possible to observe that tuned models using outperformed models using default settings (provided by LibSVM) for around 2/3 of the datasets.  However, when we consider multiple default settings (the best setting between LibSVM and optimized values), identified by the red curve, their performance values were close to the performance with tuned values.  Thus, the meta-target labelling rule considered the difference between the predictive performances with tuned hyperparameters and the best predictive performances with multiple default values.  A side effect of using multiple default values is a more class-balanced meta-dataset, increasing the proportion of meta-examples labeled with “default” use.  As a result, the imbalance rate[^17] in the meta-datasets was reduced from $\approx 2.6$ to $1.7$.

Table \[tab:meta\_datasets\] presents for each resultant meta-dataset: the $\alpha$ value used to generate the labels; the number of meta-examples, the number of meta-features and the class distribution.  It is important to observe that none of these $156$ datasets were used in a related previous study that produced optimized default setting [@Mantovani:2015c]. 

In our experimental setup, the null hypothesis of the statistical meta-label rule states that there is no significant difference between tuned and default settings.  Since we are concerned about preventing tuning when it is not necessary, a type I error is defined as labeling a meta-example as “`Tuning`” when its label is, in fact, “`Defaults`’.  Therefore, the lower the $\alpha$, the higher the probability that the improvement achieved by *tuned* values is not due to chance.  On the other hand, the higher the alpha, the lower the requirement that the performance gain by the tuning process is significant compared to default values. 

Since we are controlling the error of labeling a meta-example as “`Tuning`’, smaller $\alpha$ values will lead to a greater number of ”*default*” meta-examples.  On the contrary, the greater the alpha value, the greater the number of meta-examples labeled as“*tuning*".  As can be seen in Table \[tab:meta\_datasets\], a value of $\alpha=0.10$ implies more instances with the meta-target “`Tuning`” than when using $\alpha=0.01$. In summary, if predictive performance is more critical, the user should set the significance level as high as possible (e.g., $\alpha=0.10$).  On the other hand, if the user is concerned about computational cost, the significance level should be set to smaller values (e.g., $\alpha=0.01$).  An example of this effect can be seen in Figure \[fig:multiple\_defaults\], where the blue dots represent all the datasets where defaults should be used, i.e., tuning is not statistically significant better (for $\alpha =0.05)$.

[crcccc]{}

& & **Meta** & **Meta** &\
& & **examples** & **features** & *Tuning* & *Default*\

------------------------------------------------------------------------

SVM\_90 & $0.10$ & 156 & 80 & 102 & 54\

------------------------------------------------------------------------

SVM\_95 & $0.05$ & 156 & 80 & 98 & 58\

------------------------------------------------------------------------

SVM\_99 & $0.01$ & 156 & 80 & 94 & 62\

Experimental Setup {#subsec:setup}
------------------

Seven classification algorithms were used as meta-learners (Item 7 - Figure \[fig:mtl-framework\]): , , , , and .  These algorithms were chosen because they follow different learning paradigms with different learning biases.  All seven algorithms were applied to the meta-datasets using a $10$-fold resampling strategy and repeated $20$ times with different seeds (for reproducibility).  All the meta-datasets presented in Table \[tab:meta\_datasets\] are binary classification problems.  Thus, meta-learners’ predictions were assessed using the  performance measure, a more robust metric than for binary problems.  Moreover, also enable us to evaluate the influence of different threshold values on predictions.  Three options were also investigated at the meta-level:

-   *Meta-feature Selection*: as each meta-example is described by many meta-features, it may be the case that just a small subset of them is necessary to induce meta-models with high accuracy.  Thus, a feature selection option was added to the meta-learning experimental setup.  The method starts from an empty set of meta-features, and in each step, the meta-feature increasing the performance measure the most is added to the model.  It stops when a minimum required value of improvement ($alpha$=$0.01$) is not satisfied.  Internally, it also performs a stratified $3$-fold assessing the resultant models also according to the measure;

-   *Tuning*: since the hyperparameter values of the meta-learners may also affect their performance, tuning of the meta-learners was also considered in the experimental setup.  A simple technique was performed with a budget of $300$ evaluations and resultant models assessed through an inner stratified $3$-fold and measure.  Table \[tab:app\_space\] (Appendix \[app:mtl\_space\]) shows the hyperspace considered for tuning the meta-learners. 

-   *Data balancing*: even using the optimized default values, the classes in the meta-datasets were imbalanced.  Thus, to reduce this imbalance, the  [@Smote:2002] technique was used in the experiments. 

Some of the algorithms’ implementations selected as meta-learners use a data scaling process by default.  This is the case of the , and meta-learners.  A preliminary experiment showed that removing this option decreases their predictive performance considerably, while it does not affect the other algorithms.  When data scaling is considered for all algorithms, the performance values of , , and meta-learners were decreased.  Thus, data scaling was not considered as an option, and the algorithms used their default procedures, with which they obtained their best performance values.  Two baselines were also adopted for comparisons: a meta-dataset composed only by simple meta-features and another with data complexity ones.  Both categories of meta-features were investigated before by related studies listed in Section \[subsec:tuning\_necessity\]. 

[@lll]{}

**Element** & **Method** & **R package**\

------------------------------------------------------------------------

& & `e1071`\
& & `rpart`\
& & `randomForest`\
& & `kknn`\
& & `e1071`\
& & `gbm`\
& & `kernlab`\

------------------------------------------------------------------------

Resampling & $10$-fold & `mlr`\

------------------------------------------------------------------------

& Sequential Forward Search - $alpha=0.01$ &\
& inner 3-CV - measure &\

------------------------------------------------------------------------

& &\
& budget = 300 &\
& inner 3-CV - measure &\

------------------------------------------------------------------------

& &\
& oversampling rate = 2 &\

------------------------------------------------------------------------

& $30$ times with different seeds & -\
& from the interval $\{1, \ldots, 30\}$ & -\

------------------------------------------------------------------------

& &\
& predictions (prob) &\

------------------------------------------------------------------------

& Simple meta-features & -\
& Data complexity meta-features & -\

\[tab:mtl\_setup\]

Repositories for the coding used in this study {#subsec:repo}
----------------------------------------------

Details of the base-level tuning and meta-learning experiments are publicly available in the Studies (ids $52$ and $58$, respectively).  In the corresponding pages, all datasets, classification tasks, algorithms/flows and results are listed and available for reproducibility.  The code used for the tuning process (`HpTuning`), extracting meta-features (`MfeatExtractor`), running meta-learning (`mtlSuite`), and performing the graphical analyses (`MtlAnalysis`) are hosted at `GitHub`.  All of these repositories are also listed in Table \[tab:repo\]. 

[ll]{}

**Task/Experiment** & **Website/Repository**\

------------------------------------------------------------------------

Hyperparameter tuning code & <https://github.com/rgmantovani/HpTuning>\

------------------------------------------------------------------------

Hyperparameter tuning results & <https://www.openml.org/s/52>\

------------------------------------------------------------------------

Meta-feature extraction & <https://github.com/rgmantovani/MfeatExtractor>\

------------------------------------------------------------------------

Meta-learning code & <https://github.com/rgmantovani/mtlSuite>\

------------------------------------------------------------------------

Meta-learning results & <https://www.openml.org/s/58>\

------------------------------------------------------------------------

Graphical Analysis & <https://github.com/rgmantovani/MtlAnalysis>\

\[tab:repo\]

Results and Discussion {#sec:results}
======================

The main experimental results are described in the next subsections.  First, an overview of the predictive performance of the meta-models for the predicting task when it is worth performing tuning.  Next, different experimental setups and preprocessing techniques, such as dimensionality reduction, are evaluated.  Finally, the predictions and meta-knowledge produced by the meta-models are analyzed. 

Average performance {#sec:avg_perf}
-------------------

Figure \[fig:perf\] summarizes the predictive performance of different meta-learners for three different sets of meta-features, namely: *all*, *complex* and *simple*.  The former has all $80$ available meta-features, the *complex* set contains only $14$ data complexity measures as meta-features and the latter consists of $17$ simple and general meta-features. 

In Figure \[fig:results\], the x-axis shows the meta-learners while the y-axis shows their predictive performance assessed by the averaged over $30$ repetitions.  In addition, it shows the impact of different *alpha* ($\alpha$) levels for the Wilcoxon test for the definition of the meta-target labels.  The Wilcoxon paired-test with $\alpha=0.05$ was applied to assess the statistical significance of the predictive performance differences obtained by the meta-models with *all* meta-features, when compared to the second best approach. 

An upward green triangle () at the x-axis identifies situations where using all the meta-features were statistically better.  On the other hand, red downward triangles () show results where one of the alternative approaches was significantly better.  In the remaining cases, the predictive performance of the meta-models were equivalents. 

\

The best results were obtained by the meta-learner using data complexity (*complex*) meta-features, achieving values nearly $0.80$ for all $\alpha$ levels.  These meta-models were also statistically better than those obtained by other approaches at $\alpha=\{0.90,0.95\}$. When $\alpha=0.99$, the meta-learner using all the meta-features also generated a model with  $\approx0.8$.

When the value of $\alpha$ in the meta-label rule is reduced, predictive performances using data complexity and all the available meta-features tend to show similar distributions.  The meta-learners obtained their best values with the highest assumption ($\alpha=0.99$).  Overall, varying the $\alpha$ value did not substantially change the predictive performance of the evaluated algorithms.  In fact, few meta-examples had their meta-targets modified by the meta-rule with different values of $\alpha$. Thus, the predictions in the different scenarios are mostly the same and the performances remained similar. 

Regarding predictive performance, , , and induced accurate meta-models for the three meta-dataset variations.  The value varied in the interval $\{0.70, 0.80\}$. Even the , depending on the meta-features used to represent the recommendation problem, achieved reasonable values.  For comparison purposes, it is important to mention that both `Random` and `ZeroR`[^18] baselines obtained of $0.5$ in all these meta-datasets[^19]. 

The Friedman test [@Demvsar:2006], with a significance level of $\alpha = 0.05$, was used to assess the statistical significance of the meta-learners.  In the comparisons, we considered the algorithms’ performance across the combination of the meta-datasets and the categories of meta-features.  The null hypothesis states that all the meta-learners are equivalent regarding the performance.  When the null hypothesis is rejected, the Nemenyi post-hoc test is also applied to indicate when two different techniques are significantly different. 

Figure \[fig:results\_friedman\] presents the resultant diagram.  Algorithms are connected when there are *no* significant differences between them.  The top-ranked meta-learner was the with an average rank of $1.0$, followed by ($2.3$), ($3.2$) and ($3.6$).  They did not present statistically differences among them, but mostly did when compared with simpler algorithms: ($5.4$), ($5.7$) and ($6.5$).  Even not being statistically better than all the other choices, the was always ranked at the top regardless of the meta-dataset and meta-features. 

Although the best result was obtained using meta-features (“`complex`”), most of the meta-learners achieved their highest performance values exploring all the available meta-features.  Thus, since we want to analyze the influence of different categories of meta-features when inducing meta-models, and given the possibility of selecting different subsets from all the categories, we decided to explore all of them in the next analysis. 

Evaluating different setups
---------------------------

Due to the large difference among meta-learners results, three different setups were also evaluated to improve their predictive performances and enable a comprehensive analysis of the investigated alternatives:

-   `featsel` - meta-feature selection via  [@mlr:2016];

-   `tuned` - tuning of the meta-learners using a simple technique;

-   `smote`: dataset balancing with  [@Smote:2002]. 

They were compared with the original meta-data with no additional process (`none`), which is the baseline for these analyses.  These setups were not performed at the same time to avoid overfitting, since the meta-datasets have $156$ meta-examples and, depending on their combinations, three levels of would be used to assess models.  For example: if feature meta-selection and tuning were enabled at the meta-level, one would be used for meta-feature selection, one for tuning and another to assess the resulting models. 

\
\

Figure \[fig:perf\] summarizes the main aspects of these experimental results.  The top figure shows the average values for each experimental setup considering all the meta-learners and the $\alpha$ levels.  The and meta-learners do not have any tunable .  Thus, their results in this figure are missing for the `tuned` setups (with and without ).  Similarly to Figure \[fig:results\], the statistical analysis is also presented.  Every time an upward green triangle is placed at the x-axis, the raw meta-data (`none`) generated results statistically better than using the best of the experimental setups evaluated.  On the other hand, red triangles indicate when tuning, meta-feature selection or could statistically improve the predictive performance of the meta-models.  In the remaining cases, the meta-models were equivalent. 

Despite the different setups evaluated, is still the best meta-learner for all $\alpha$ scenarios.  It is followed by and versions using .  Depending on the experimental setup, and also presented good predictive performances.  Regarding the tuning (`tuned`) of the meta-learners, only for the performances slightly improved for all the alpha values.  Using just resulted in improved results for , and meta-learners.  In general, it produced small improvements, but most of them were statistically significant.  When used with tuning or meta-feature selection, it affected the algorithms in different ways: for and , the performance improved; for , and there was no benefit; the other algorithms were not affected by its use.  The low gain obtained using may be due to the fact that data imbalance was already reduced using the optimized defaults when defining meta-targets. 

Using meta-feature selection (`featsel`) deteriorated the performance of the , , and meta-learners.  On the other hand, it clearly improved the , and performances for most cases.  benefited from using a subset of meta-features to maximize the importance of more relevant meta-features.  For and , selecting a subset of the attributes reduced the presence of noise and irrelevant attributes.  Furthermore, it is important to observe that the meta-models induced with the selected features presented the highest standard deviation between the setups (light area along the curve).  A possible reason is the different subsets selected every time meta-feature selection is performed for the $30$ repetitions. 

Additionally, Figure \[fig:auc\_rk\] presents a ranking with all the combinations of meta-learners and experimental setups.  At the x-axis, they are presented in ascending order according to their average ranking for the three scenarios ($\alpha$ values), shown at the y-axis.  The more red the squares, the lower the ranking, i.e., the better the results. 

As previously reported, with no additional option was the best-ranked method, followed by its smoted versions.  The , and versions are in the next positions.  The Friedman test with a significance level of $\alpha=0.05$ was also used to assess the statistical significance of the meta-learners when using different experimental setups in different meta-datasets.  Figure \[fig:cd\_rks\] shows the resultant diagram.  Results are quite similar to those reported in Figure \[fig:results\_friedman\]: the algorithm was the best algorithm with an average ranking of $1.05$, and is statistically better than most of the meta-learners, except for and . 

Since there were no improvements considering the maximum values achieved so far, and the results from the meta-learners were still the best-ranked, the next subsections will analyze the relative importance of the meta-features according to the final induced models. 

Importance of meta-features {#subsec:rf}
---------------------------

From the induced meta-models, the relative importance of the meta-features based on the Gini impurity index used to calculate the node splits [@Breiman:2001].  Figure \[fig:rf-mfeat\] shows the average relative importance of the meta-features obtained from the meta-models.  The relative importance is shown for the experiments considering all meta-features and $\alpha=0.05$ (middle case).  At the x-axis, meta-features are presented in decreasing order according to their average relative importance values.  From this point, anytime a specific meta-feature is mentioned we present its name with a prefix indicating its category (according to Table \[tab:mf-categories\]). 

[ \[width=\] [05\_RF\_OptDfs\_new.png]{} \[fig:auc\_gini\] ]{}

Since no negative value (negative relative importance) was obtained, no meta-feature was discharged to build meta-models.  It also shows that a large number of meta-features were relevant to the induction of the meta-models, a possible reason for why meta-feature selection produced worse results for most of the meta-learners. 

The most important meta-feature was a landmarking meta-feature: “`LM.stump_sd`”, which describes the standard deviation of the number of examples correctly classified by a decision stump.  It measures the complexity of the problem considering its simplicity.  The second most important was a simple meta-feature: “`SM.classes_min`”, which measures the minority class size.  The third was also a simple meta-feature: “`SM.classes_sd`”, which describes the standard deviation of the number of examples per class.  These meta-features together strongly indicate that for , the most important meta-features are related with class imbalance.  A rule extracted from a model induced by states that if the dataset is imbalanced, it is better to use default for .  The other important meta-features were:

-   “`IN.nClEnt`” and “`IN.mutInf`”: these are information-theoretical meta-features.  While the first describes the class entropy for a normalized base level dataset, the second measures the mutual information, a reduction of uncertainty about one random feature given the knowledge of another;

-   “`CN.betweenness`”: betweenness centrality is a meta-feature derived from complex networks that measures, for a set of vertex and edges, the average number of shortest paths that traverses them.  The value will be small for simple datasets, and high for complex datasets;

-   “`DC.l1`” and “`DC.t2`”: these are data complexity meta-features.  While the first measures the minimum of an error function for a linear classifier, the second measures the average number of points per dimension.  These features are related with the class separability (l1), and the geometry of the problem’s dimension (t2);

-   “`SM.dimension`”: this is a simple meta-feature that measures the relation between the number of examples and attributes in a dataset;

-   “`CN.maxComp`”: this is another complex-network meta-feature.  It measures the maximum number of connected components in a graph.  If a dataset presents a high overlapping of its classes, the graph will present a large number of disconnected components, since connections between different classes are pruned. 

Among the most important, there are meta-features from different categories (simple, data complexity, complex-networks and from information-theory).  Complex-network measures describe data complexity regarding graphs and indicate how sparse the classes are between their levels.  Data complexity meta-features try to extract information related to the class separability.  The stump meta-feature works along the same lines, trying to identify the complexity of the problem by simple landmarking.  The information-theoretical meta-features indirectly checks how powerful the dataset attributes are to solve the classification problem. 

Although summarized rules cannot be obtained from meta-models, the analysis of meta-features importance provides some useful information.  For instance, dataset characteristics such as the data balancing, class sizes, complexity and linearity were considered relevant to recommend when tuning is required. 

Linearity Hypothesis {#subsec:when}
--------------------

The previous sections, in particular the meta-analysis, suggest that linearity is a key aspect to decide between the recommendation of default or tuned values.  Experimental results indicate that default values might be good for classification tasks with high linear separability.  As a consequence, tuning would be required for tasks with complex decision surfaces, where would need to find irregular decision boundaries. 

\

In order to investigate this hypothesis, a linear classifier was also evaluated in all the available $156$ datasets using the same base-level experimental setup described in Table \[tab:hp\_setup\].  If the linearity hypothesis is true, the performance difference between and the linear classifier in meta-examples labeled as “`Defaults`” would be smaller than or equal to the meta-examples labeled as “`Tuning`’. 

Figure \[fig:perf\_diff\] shows the performance differences obtained in all the datasets at the base-level.  Datasets at the x-axis are split based on their meta-target labels, “`Tuning`’, left side, in black, and “`Defaults`”, right side, in red.  Despite some outliers, the performance differences for “`Tuning`” meta-examples are in general much higher than those for the “`Defaults`” meta-examples.  Thus, the observed patterns support the linearity hypothesis. 

In [@Leite:2012], the authors proposed a set of “” meta-features based on the pairwise performance difference of simple landmarking algorithms.  This new data characterizations schema is used to train meta-learning based on the algorithm.  The patterns observed in Figure \[fig:perf\_diff\] follow the same principle, presenting a new alternative to characterize base-level datasets.  Following this proposal, $10$ new relative landmarking meta-features were generated based on five landmarking algorithms: , , , and .  These new meta-features are described in Table \[tab:mfeats\_2\] in Appendix \[app:features\]. 

The same meta-analysis described in Section \[subsec:rf\] was performed, adding the relative landmarking meta-features to the meta-datasets.  These experiments pointed out how useful the new meta-features are for the recommendation problem.  Figure \[fig:newFeat\] shows the relative importance values of the meta-features averaged in $30$ executions.  The relative importance of these new meta-features are highlighted in red, while the simple landmarking is shown in blue. 

Two of the relative landmarking meta-features are placed in the top-10 most important meta-features: `RL.diff.nn.lm` ($1ˆ{st}$), and `RL.diff.svm.lm` ($3ˆ{rd}$); another two measures are in the top-20 - `RL.diff.stump.lm` and `RL.diff.stump.lm`; and all of them depend directly on the linear classifier performance.  It is also important to mention that simple landmarking meta-features performed, in general, worse than relative landmarking.  All these relative importance plots show evidence that the linearity hypothesis is true, and at least one characteristic that defines the need of tuning for is the linearity of the base-level classification task. 

Overall comparison
------------------

Given the potential shown by the relative landmarking meta-features, they were experimentally evaluated in combination with the meta-features previously evaluated as most important.  Complex network (`cnet`) meta-features were included because they were ranked between the most important descriptors (as shown in Subsection \[subsec:rf\]).  Simple and data complexity (`complex`) meta-features were the other two approaches evaluated in the related studies listed in Section \[subsec:tuning\_necessity\]. 

\

Figure \[fig:comparison\] presents a comparison between the main experimental setups considering the addition of the relative landmarking (`relativeLand`) meta-features.  The left chart of figure \[fig:mfeat\_relativeLand\] shows performance values obtained for each of the original setups, while the chart on the right presents setup performances when relative landmarking meta-features were included.  This figure shows that the use of relative landmarking meta-features improved all the setups where they were included.  At least three different setups used by were higher than the performance value obtained in the initial experiments.  The setup considering simple and relative landmarking meta-features induced the best meta-models for , and .  The and meta-learners obtained the best predictive performance using data complexity and relative landmarking meta-features and the same occurred for and for “relativeLand” set. 

Figure \[fig:mfeat\_significance\] compares the best setup from Figure \[fig:mfeat\_relativeLand\]: “`simpleRelativeLand`”, which uses both simple and relative landmarking meta-features, with the the baselines from Figure \[fig:results\], using“`simple`” and data complexity (“`complex`”) meta-features, often explored in related studies (see Table \[tab:svm\_related\]). 

In this figure, the x-axis shows the different meta-learners, while the y-axis shows their predictive performance assessed by the AUC averaged over $30$ repetitions.  The Wilcoxon paired-test with $\alpha$ = 0.05 was applied to assess the statistical significance of these results.  An upward green triangle () at the x-axis identifies situations where the use of “`simpleRelativeLand`” was statistically better than using the baselines.  In the same figure, the red downward triangles () indicate when the use of baselines was significantly better.  In the remaining cases, the predictive performance of the meta-models were equivalents. 

Overall, the meta-models induced with “`simpleRelativeLand`” meta-features were significantly better than those induced with baseline meta-features for most of the meta-learners: , , and obtained superior values.  Furthermore, the best meta-learner (RF) also significantly outperformed our previous results.  The baselines produced the best meta-models for only two algorithms: and .  For the algorithm, the different setups did not present any statistically significant difference. 

Analysis of the predictions
---------------------------

A more in-depth analysis of the meta-learner predictions can help to understand their behavior.  Figure \[fig:pred\_all\] shows the misclassifications of the meta-learners considering their best experimental setups.  The top chart (Fig.   \[fig:preds\]) shows all the individual predictions, with the x-axis listing all the meta-examples and y-axis the meta-learners.  In this figure, “`Defaults`” labels are shown in black and “`Tuning`” labels in gray.  The top line in the y-axis, “`Truth`” shows the truth labels of the meta-examples, which are ordered according to their truth labels.  The bottom line (“\*”) shows red points for meta-examples misclassified by all meta-learners. 

\

In the tuning recommendation task, “Defaults” is defined as the positive class and “`Tuning`” as the negative class.  Therefore, a is a wrong recommendation to perform tuning on , and is a wrong recommendation to use default values.  While a reduction in can decrease the computational cost, a reduction in can improve predictive performance. 

Algorithms following different learning biases present different prediction patterns and this can be observed in Figure  \[fig:pred\_all\].  Usually, most meta-examples are correctly classified (a better performance than the baselines).  Besides, the following patterns can be observed:

-   and minimize the rate, correctly classifying most of the meta-examples as “`Defaults`”.  However, they misclassified many examples from the “`Tuning`” class, penalizing the overall performance of the recommender system;

-   , and minimized the rate, correctly classifying most of the meta-examples requiring tuning.  However, they tended to classify the meta-examples in the majority class;

A more balanced scenario is provided by the meta-models, which presented the best predictive performance.  Although it was not the best algorithm for each class individually, it was the best when the two classes were considered. 

[clrrrrcccc]{}

**Nro** & **Name** & **id** & **D** & **N** & **C** & **P** & **Def (sd)** & **Tun (sd)** & **Label**\

------------------------------------------------------------------------

17 & jEdit\_4.0\_4.2 & 1073 & 8 & 274 & 2 & 0.96 & 0.73 & 0.73 & `Defaults`\
36 & banknote-authentication & 1462 & 4 & 1372 & 2 & 0.80 & 0.99 & 0.99 & `Tuning`\
78 & autoUniv-au7-500 & 1554 & 12 & 500 & 5 & 0.22 & 0.29 & 0.31 & `Tuning`\
97 & optdigits & 28 & 62 & 5620 & 10 & 0.97 & 0.99 & 0.99 & `Tuning`\

\[tab:misclassified\]

Table \[tab:misclassified\] lists all the datasets misclassified by all the meta-learners as indicated in Figure \[fig:preds\].  Meta-examples with ids $17$ (“`Defaults`”) and $78$ (“`Tuning`”) were corrected labeled by the statistical meta-rule, and therefore the misclassification may have occurred due to the lack of the descriptive ability of meta-features or noise in the meta-dataset.  The other two meta-examples ($36$, $97$) were both labeled as “`Tuning`” but the statistical difference indicated is very small in terms of performance, and may indicate a limitation of the current meta-target rule criteria. 

Projecting performances at base-level
-------------------------------------

This section assesses the impact of the choices made by the meta-learners at the base-level.  It also analyses and discusses the reduction of runtime when using the proposed meta-learning recommender system.  Figure \[fig:final\] shows the predictive performance of at the base-level using the method (“`Tuning`’ or “`Defaults`’) selected by the meta-learners to define their values.  The best meta-learners identified in the previous sections were compared with three simple baselines: a model that always recommends tuning (`Tuning`), a model that always recommends the use of defaults (`Defaults`), and a model that provides random recommendations (`Random`). 

Figure \[fig:perf\_projec\] shows a scatter plot with the projected performance and runtime value averaged in all the base-level datasets.  Performing always the tuning had the highest average value but was also the most expensive approach.  On the other hand, always using default values is the fastest approach, but with the lowest average value.  The proposed meta-models are above `Random` and `Defaults` baselines, performing close to the `Tuning` baseline but with lower average runtime costs. 

\

The Friedman test, with a significance level of $\alpha = 0.05$, was also used to assess the statistical significance of the base-level predictions.  The null hypothesis is that all the meta-learners and baselines are equivalent regarding the average predictive performance.  When the null hypothesis is rejected, the Nemenyi post-hoc test is applied to indicate when two different techniques are significantly different.  Figure \[fig:friedman\_test\] presents the diagram.  Techniques are connected when there is *no* significant differences between them. 

Overall, the approach always using default (`Defaults`) is ranked last, followed by the `Random` baseline.  Almost all meta-learners are significantly better than both and are equivalent to `Tuning`, which always performs tuning.  In this figure, although the meta-model is considered the best, it was not ranked first.  The first was the meta-model.  This occurred because it most often recommended the use of tuned settings, which was reflected in its performance at the base-level.  With many datasets at the base-level, it can be pointed out that the overall gain is diluted between them.  Even so, the meta-learners could considerably reduce the computational costs related to tuning, maintaining a high predictive performance. 

Besides, it can be observed in Figure \[fig:perf\_projec\] that tuning hyperparameters for just one dataset will take on average two days.  The most costly datasets (with a high number of features or examples) took almost 10 days to finish all the 10 tuning repetitions (seeds) even paralleling the jobs in a high performance cluster.  Regarding the meta-feature extraction, the same datasets will take at most 10 minutes, specially because of some mathematical operations used by  meta-features.  For most meta-features, the time taken to characterize a dataset is less than $30$ seconds.  Thus, during the prediction phase with the induced meta-model, the computational cost of extracting the characteristics of a new dataset is irrelevant compared to the computational cost of the tuning process.  We think this is an important argument in favor of using our system, which is used in practical scenarios. 

A note on the generalization {#sec:generalization}
----------------------------

Although the main focus of the paper is to investigate the hyperparameter tuning problem, we also conducted experiments for predicting the need for tuning .  These experiments aim to provide more evidence that the proposed method can be generalized.  Once the meta-knowledge is extracted, the system is able to induce meta-models to any supervised learning algorithm.  In particular, we investigated the tuning of the `J48` algorithm, a `WEKA`[^20] implementation for the Quinlan‘s C4.5  induction algorithm [@Quinlan:1993], and one of the most popular algorithms.  The tree models were induced using the `RWeka`[^21].  We also expanded the meta-knowledge by performing additional experiments to cover the same datasets explored with .  In total, we obtained results performing the tuning of the J48 algorithm in $165$ datasets.  The tuning processes followed the experimental setup described in Table \[tab:hp\_setup\] with some differences:

-   the J48 hyperparameter space has nine hyperparameters[^22];

-   a budget size of $900$ evaluations was adopted in the experiments with trees.  It is greater than for because of the greater search space of J48;

-   the tuned hyperparameter results were compared with those obtained from the `J48` `RWeka`/`WEKA` default settings. 

Table \[tab:j48\_datasets\] in \[app:j48\_space\] presents the main characteristics of the meta-datasets generated with J48 tuning experiments.  Class distribution columns (*Tuning* and *Default*) show values which indicate a different *hyperparameter profile*[^23] compared to that observed in experiments with : most of the meta-examples are labelled as “`Default`”, i.e., tuning did not statistically improve the algorithm performance in two thirds of the datasets.  Here, we present results just on the meta-dataset using $\alpha$ = 0.05 for the statistical labeling rule.  However, results obtained with different $\alpha$ values were quite similar. 

\

The predictive performance of the meta-learners considering different categories of meta-features are summarized in Figure \[fig:j48\_mtl\_auc\].  The best results were obtained using the , and meta-learners.  They achieved their best predictive performances using preferably “all” the available meta-features, with values between $\:=(0.67,0.72)$. It may be due to the fact that predictions tend toward a specific class (Defaults) since these meta-datasets are imbalanced.  Overall, the best meta-model was obtained by the algorithm.  However, when considering all the possible scenarios (different $\alpha$ values), there was no statistical difference among the top ranked meta-learners: the , , , and algorithms (see Figure \[fig:j48\_friedman\_mfeats\]).  Furthermore, in general, the complete set of meta-features provided the best results for most algorithms. 

Figure \[fig:j48\_mtl\_setups\] shows the average values considering different experimental setups.  The `NB` and `LR` algorithms do not have any tunable hyperparameters.  Consequently, their results for “`tuned`” setups are missing in the chart.  In general, when considering values obtained in the original meta-dataset, improvements were obtained only for and applying .  For the other algorithms, best meta-learners were still induced without any additional process.  Statistical comparisons highlight the performance of the , and algorithms (see Figure \[fig:j48\_friedman\_setups\]). 

We also evaluated the potentiality of the meta-features in the J48 tuning recommendation problem.  However, differently than reported in Section \[subsec:when\], they worsened most of the meta-models.  Linearity is not a key aspect in the J48 tuning problem, which further reinforces the results obtained from (see Section \[subsec:when\]). 

Reproducing the analysis in the J48 tuning problem (see Section \[subsec:rf\]), the most important meta-feature was the data complexity measure “`DC.f4`”.  This meta-feature describes the collective attribute efficiency in a dataset.  The second was a simple meta-feature: “`SM.abs_cor`”, a metric that measures the linear relationship between two attributes.  This value is averaged in all pairs of attributes in the dataset.  The top-3 is completed with another data complexity meta-feature: “`DC.f3`”, which describes the maximum individual attribute efficiency.  The two meta-features measure the discriminative power of the dataset’s attributes, while the absolute correlation verifies if the information provided by attributes is not redundant.  These most important meta-features suggest that if a dataset has representative attributes, default values are robust to solve it.  Otherwise, the J48 tuning is recommended. 

\

Figure \[fig:dt\_projected\_results\] shows J48 meta-level predictions projected overall base-level datasets.  Mostly, but not all of the meta-examples are labeled with “`Defaults`”.  The induced meta-models depicted in Figure \[fig:j48\_projections\] are above the “`Defaults`” baseline, but relatively close to the `Random` and `Tuning` ones.  The average values of all the approaches are very close, even considering all the baselines.  This can be noted by the scale on the y-axis.  It is explained by the small improvements obtained at the base-level tuning processes; they were relatively small if compared with those obtained with . 

However, all the meta-models have a lower average runtime compared with `Random` and `Tuning` baselines.  Most of the meta-models are better ranked than baselines, but, overall, there are no statistical differences among the evaluated approaches (Figure \[fig:friedman\_test\_j48\]).  Even so, it is important to highlight that meta-learners could also considerably reduce the computational cost related to tuning, keeping the predictive performance in the dataset collection. 

Conclusions {#sec:conclusions}
===========

This paper proposed and experimentally investigated a framework to predict when to perform tuning.  To do this, 156 different datasets publicly available at were used.  The predictive performances of induced with tuning (by a simple ) and with default values were compared and used to design a recommender system.  The default values were provided by the `e1071` R package), and by optimized common settings from [@Mantovani:2015c].  Different experimental setups were analyzed with different sets of meta-features.  The main findings are summarized next. 

Tuning prediction
-----------------

The main issue investigated in this paper was whether it is possible to accurately predict when tuning can improve the predictive performance of , when compared to the use of default values.  If so, this can reduce the processing cost when applying to a new dataset.  According to experimental results, using and as meta-learners, this prediction is possible with a predictive accuracy of =$0.798$.

Three significance levels ($\alpha=\{0.01, 0.05, 0.10\}$) were used with the Wilcoxon test to define the meta-target, which indicates whether it is better to use tuning or default values.  Different sets of meta-features were evaluated, and meta-models using all the available meta-features obtained the best results, regardless the $\alpha$ value considered.  However, the complex set of meta-features resulted in high predictive performance for most of the investigated meta-learners.  Different experimental setups were also evaluated at the meta-level, but improvements were observed, and in a few cases, only when or meta-feature selection were used.  Thus, the best setup was to use raw meta-data and meta-learners with default values. 

An analysis of the meta-models show that most meta-features actively contributed to predictions.  This explains the decrease in performance when using meta-feature selection in most of the algorithms.  Among the $10$ meta-features ranked as most important, there are meta-features from different sets.  Each ranked meta-feature describes different characteristics of the problem, such as data imbalance, linearity, and complexity. 

This paper also investigated the hypothesis that the linear separability level could be an important meta-feature to be used by the recommender system.  Meta-features based on relative landmarking were used to measure the linear separability degree.  In the experiments performed, this meta-feature was shown to play an important role in the recommender system prediction.  Three meta-learners had their best performance values using a combination of simple and relative landmarking meta-features. 

Using two different default settings maximized the number of default wins, reducing the imbalance rate at the meta-datasets.  In addition to presenting the best predictive performance, , by the frequency of meta-features in their trees, provided useful information regarding when default settings are suitable. 

We also performed experiments for the J48 tuning recommendation problem aiming to show the ability of generalization of the recommender system.  Results showed that different to , where the linearity was important to recommend the use of default settings, the most important meta-features suggest that if a dataset has representative attributes, default settings are robust to solve it. 

In fact, our extensive experiments suggest that the guideline depends on the algorithm used to induce the meta-model.  If we use a white box algorithm, such as , we can use the meta-features in the root of the trees (and nodes close to the root) to explain when to tune.  The high predictive performance of the algorithm indicates that the induced models were able to find a good hypothesis for situations where tuning is necessary, in both cases (for and J48). 

Linking findings with the literature
------------------------------------

Two of the related studies in the literature [@Mantovani:2015b; @Ridd:2014] used meta-models based on decision trees to interpret their predictions.  However, in the current results, trees were among the worst meta-learners considering all the experimental setups analyzed.  Thus, in this study, the meta-analysis performed was based on the meta-learner, extracting the average of the Gini index from the meta-features provided by the inner meta-models. 

Meta-feature selection was also evaluated in [@Ridd:2014].  The authors explored a method and reported the “`nn`” meta-feature[^24] as the most important.  However, the results reported here were not improved by meta-feature selection.  In fact, it decreased the performance of most meta-models, as shown in Figure \[fig:perf\](a).  Meta-feature selection was also tried with filter methods, but the results were even worse than using .  For this reason, it was not reported in this study.  In addition, “`nn`” meta-features did not appear among the top-20 most important meta-features computed by . 

Our experimental results show that using meta-features from different categories have improved the predictive performance of the meta-learners for different setups.  The most important meta-features were ”`SM.classes_min`” and ”`LM.stump_sd`”[^25].  In  [@Mantovani:2015b], which only used meta-features from simple and data complexity sets, “`SM.classes_max`” and “`SM.attributes`” were reported as the most important meta-features.  The first describes the percentage of examples in the majority class.  The second is the number of predictive attributes in a dataset.  Although these studies disagree about the relative importance order of these meta-features, both extract information related to the same characteristics: data complexity and dimensionality. 

The tuning investigated in [@Sanders:2017] “*assumed that tuning is always necessary*”, and therefore focused on the improvement prediction as a regression task.  They obtained tuning results for less than half of the datasets (111/229) in the base-level when dealing with .  In addition, their meta-models were not able to predict the improvement for , not providing any valid conclusion about the problem. 

Main difficulties
-----------------

During the experiments, there were several difficulties to generate the meta-knowledge.  The process itself is computationally expensive, since a lot of tuning tasks must be run and evaluated in a wide range of classification tasks.  Initially, a larger number of datasets were selected, but some of them were extremely expensive computationally speaking for either tuning or the extraction of meta-features.  A *walltime* of $100$ hours was considered to remove high-cost datasets. 

The class imbalance at the meta-level was another problem faced in the experiments.  To deal with this problem, the optimized default settings [@Mantovani:2015c] were added to the meta-dataset, increasing the number of meta-examples in the default meta-target.  Besides the addition of relative landmarking set, some meta-examples were never correctly classified.  This points out the need to define specific meta-features for some problems. 

Future work
-----------

Some findings from this study also open up future research directions.  The proposed MtL recommender system could be extended to different algorithms, such as neural networks, another decision tree induction algorithms and ensemble-based techniques.  It could also be used to support tuning decision in different tasks, such as pre-processing, regression and clustering.  It could even be used to define whether to tune for more than one task, in a pipeline or simultaneously. 

It would also be a promising direction to investigate the need of new meta-features to characterize data when dealing with data quality problems, for instance imbalancing measures, due to their influence in the quality of the induced meta-models.  Besides, a multicriteria objective function could replace the current meta-label rule, weighing predictive performance, memory, and runtime.  Another possibility would be to explore the use of ensembles as meta-models, given the complementary behavior of some of the algorithms studied here as meta-learners. 

The code used in this study is publicly available, easily extendable and may be adapted to cover several other algorithms.  The same may be said of the analysis, also available for reproducibility.  All the experimental results generated are also available at correspondent studies web pages, from which they can be integrated and reused in different systems.  This framework is expected to be integrated to , so that the scientific community can use it. 

Acknowledgments {#acknowledgments .unnumbered}
===============

The authors would like to thank CAPES and CNPq (Brazilian Agencies) for their financial support, specially for grants \#2012/23114-9, \#2015/03986-0 and \#2018/14819-5 from the S[ã]{}o Paulo Research Foundation (FAPESP). 

 {#section .unnumbered}

[50]{} natexlab\#1[\#1]{}\[1\][`#1`]{} \[2\][\#2]{} \[1\][\#1]{} \[1\][[](http://dx.doi.org/#1)]{} \[1\][[](pmid:#1)]{} \[2\][\#2]{} , . . ;():. , , . .  In: . .  p. . , . . ;:. , , , , , , , . . ;():. , , , . .  ed. , . , . .  In: , , , , editors. . ; .  p. . . . ;():. , , , . .  In: . ; .  p. . , , , , , , , , . . ;:. , . . ;:. , , , . . ;():. . . ;:. , , , , . . ;():. , , , , , . .  In: , , , , , editors. . ; .  p. . , , . . ;:. , , , , . . ;():. , . . ;():. , , , , . . ;:. , , . . ; ; . . , , , . . ;():. , , , , . . ;:. , , , . ;():. , , . .  In: . .  p. . , , , , . . ;:. , , , , . . ;():. , , , , . .  In: . ; .  p. .  <http://dx.doi.org/10.1109/BRACIS.2016.018>. . , , , , . .  In: . ; .  p. . , , , , . .  In: . ; .  p. . , , , . .  In: , , , , editors. ;  volume of **; .  p. . , . .  In: . .  p. . , , . .  In: . .  p. . , , , . . ;():. , . .  In: . ; .  p. . , , , , , . ; : .  p. . , , . .  In: . : ; .  p. . , , , . .  In: .  volume ; .  p. . . . : , . , , . .  In: . ;  KI’11; .  p. . , , . . ;:. , , , , . . ;():. , . .  In: , , , , editors. . .  p. . , . .  In: , , , , , editors. . ; .  p. . , , . .  In: , , , , editors. . ; .  p. . , . .  In: . ;  SAC’06; .  p. . , , . . ;():. , , , . .  In: . .  p. . , , , . . ;():. . . , . , , , . . ;():. , , . . ;():. 

List of Meta-features used in experiments {#app:features}
=========================================

[lll]{}

& **Acronym** & **Description**\

& classes & Number of classes\
& attributes & Number of attributes\
& numeric & Number of numerical attributes\
& nominal & Number of nominal attributes\
& samples & Number of examples\
& dimension & samples/attributes\
& numRate & numeric/attributes\
& nomRate & nominal/attributes\
& symbols (min, max, mean, sd, sum) & Distributions of categories in attributes\
& classes (min, max, mean, sd) & Classes distributions\

------------------------------------------------------------------------

& sks & Skewness\
& sksP & Skewness for normalized dataset\
& kts & Kurtosis\
& ktsP & Kurtosis for normalized datasets\
& absC & Correlation between attributes\
& canC & Canonical correlation between matrices\
& frac & Fraction of canonical correlation\

------------------------------------------------------------------------

& clEnt & Class entropy\
& nClEnt & Class entropy for normalized dataset\
& atrEnt & Mean entropy of attributes\
& & Mean entropy of attributes for\
& & normalized dataset\
& jEnt & Joint entropy\
& mutInf & Mutual information\
& eqAtr & clEnt/mutInf\
& noiSig & (atrEnt - mutInf)/MutInf\

------------------------------------------------------------------------

& nodes & Number of nodes\
& leaves & Number of leaves\
& nodeAtr & Number of nodes per attribute\
& nodeIns & Number of nodes per instance\
& leafCor & leave/samples\
& lev (min, max, mean, sd) & Distributions of levels of depth\
& bran (min, max, mean, sd) & Distributions of levels of branches\
& att (min, max, mean, sd) & Distributions of attributes used\

------------------------------------------------------------------------

& nb & Naive Bayes accuracy\
& stump (min, max, mean, sd) & Distribution of decision stumps\
& stMinGain & Minimum gain ratio of decision stumps\
& stRand & Random gain ratio of decision stumps\
& nn & 1-Nearest Neighbor accuracy\

[lll]{}

& **Acronym** & **Description**\

& f1 & Maximum Fisher’s discriminant ratio\
& f1v & Directional-vector maximum Fisher’s discriminant ratio\
& f2 & Overlap of the per-class bounding boxes\
& f3 & Maximum feature efficiency\
& f4 & Collective feature efficiency\
& l1 & Minimized sum of the error distance of a linear classifier\
& l2 & Training error of a linear classifier\
& l3 & Nonlinearity of a linear classifier\
& n1 & Fraction of points on the class boundary\
& n2 & Ratio of average intra/inter-class NN distance\
& n3 & leave-one-out error rate of the 1-NN classifier\
& n4 & Nonlinearity of the 1-NN classifier\
& t1 & Fraction of maximum covering spheres\
& t2 & Average number of points per dimension\

------------------------------------------------------------------------

& edges & Number of edges\
& degree & Average degree of the network\
& density & Average density of the network\
& maxComp & Maximum number of components\
& closeness & Closeness centrality\
& betweenness & Betwenness centrality\
& clsCoef & Clustering Coefficient\
& hubs & Hub score\
& avgPath & Average path length\

------------------------------------------------------------------------

& diff.svm.lm & performance(SVM) - performance(Linear)\
& diff.svm.nb & performance(SVM) - performance(NB)\
& diff.svm.stump & performance(SVM) - performance(Decision Stump)\
& diff.svm.nn & performance(SVM) - performance(1-NN)\
& diff.nn.lm & performance(1-NN) - performance(Linear)\
& diff.nn.stump & performance(1-NN) - performance(Decision Stump)\
& diff.nn.nb & performance(1-NN) - performance(NB)\
& diff.nb.stump & performance(NB) - performance(Decision Stump)\
& diff.nb.lm & performance(NB) - performance(Linear)\
& diff.stump.lm & performance(Decision Stump) - performance(Linear)\

Hyperparameter space of the meta-learners used in experiments {#app:mtl_space}
=============================================================

[cclcccc]{}

**Algo** & **Symbol** & **hyperparameter** & **Range** & **Type** & **Default** & **Package**\

------------------------------------------------------------------------

& cp & complexity parameter & $(0.0001, 0.1)$ & real & $0.01$ &\

------------------------------------------------------------------------

& & minimum number of instances in a & & & &\
& & node for a split to be attempted\

------------------------------------------------------------------------

& minbucket & minimum number of instances in a leaf & $[1,50]$ & integer & $7$ &\

------------------------------------------------------------------------

& & maximum depth of any node of & & & &\
& & the final tree\

------------------------------------------------------------------------

GP & sigma & width of the Gaussian kernel & $[2^{-15}, 2^{15}]$ & real & - & `kernlab`\

------------------------------------------------------------------------

& k & kernel & Gaussian & - & - &\

------------------------------------------------------------------------

& C & regularized constant & $[2^{-15},2^{15}]$ & real & 1\

------------------------------------------------------------------------

&$\gamma$ & width of the Gaussian kernel & $[2^{-15}, 2^{15}]$ & real & ${{}^{1}\!/_{N}}$ &\

------------------------------------------------------------------------

& ntree & number of trees & $[2^0,2^{10}]$ & integer & 500 &\

------------------------------------------------------------------------

& nodesize & minimum node size of the decision trees & $\{1,20\}$ & integer & 1 &\

------------------------------------------------------------------------

KNN & k & number of nearest neighbors & $\{1,50\}$ & integer & 7 & `kknn`\

------------------------------------------------------------------------

NB & - & - & - & - & - & `e1071`\

------------------------------------------------------------------------

LR & - & - & - & - & - & `gbm`\

J48 hyperparameter space and meta-datasets used in experiments from Section \[sec:generalization\] {#app:j48_space}
==================================================================================================

[clcccc]{}

**Symbol** & **Hyperparameter** & **Range** & **Type** & **Default** & **Conditions**\

------------------------------------------------------------------------

C & pruning confidence & $(0.001, 0.5)$ & real & 0.25 & R = False\

------------------------------------------------------------------------

M & minimum number of instances in a leaf & $[1,50]$ & integer & 2 & -\

------------------------------------------------------------------------

& number of folds for reduced & & & &\
& error pruning & & & &\

------------------------------------------------------------------------

O & do not collapse the tree & {False,True} & logical & False & -\

------------------------------------------------------------------------

R & use reduced error pruning & {False,True} & logical & False & -\

------------------------------------------------------------------------

B & use binary splits only & {False,True} & logical & False & -\

------------------------------------------------------------------------

S & do not perform subtree raising & {False,True} & logical & False & -\

------------------------------------------------------------------------

& Laplace smoothing for predicted & & & & -\
& probabilities & & & &\

------------------------------------------------------------------------

& do not use MDL correction for & & & & -\
& info gain on numeric attributes & & & &\

[crcccc]{}

& & **Meta** & **Meta** &\
& & **examples** & **features** & *Tuning* & *Default*\

------------------------------------------------------------------------

J48\_90 & $0.10$ & 165 & 80 & 63 & 102\

------------------------------------------------------------------------

J48\_95 & $0.05$ & 165 & 80 & 57 & 108\

------------------------------------------------------------------------

J48\_99 & $0.01$ & 165 & 80 & 52 & 113\

List of abbreviations used in the paper {#app:glossary}
=======================================

[^1]: Google Cloud AutoML - <https://cloud.google.com/automl/>

[^2]: Microsoft Custom Vision - <https://www.customvision.ai>

[^3]: The code is available in `Github` repositories, while experimental results are available on  [@Vanschoren:2014] study pages.  These links are provided in Table \[tab:repo\] at Subsection \[subsec:repo\]. 

[^4]: A note on the generalization of the proposal is presented in Section \[sec:generalization\]

[^5]: The `e1071` package was used to implement .  It is the LibSVM [@Chang:2011] interface to the R environment. 

[^6]: More details about dataset versions can be found in the paper [@Vanschoren:2014] and documentation page: <https://docs.openml.org/#data>. 

[^7]: <https://www.openml.org/s/52/data>

[^8]: <https://github.com/openml/openml-r>

[^9]: <https://github.com/mlr-org/mlr>

[^10]: LibSVM default values can be consulted at <https://www.csie.ntu.edu.tw/~cjlin/libsvm/>

[^11]: <http://www.cs.ubc.ca/labs/beta/Projects/autoweka/>

[^12]: <https://github.com/automl/auto-sklearn>

[^13]: These findings go towards what was previously described in  [@Bergstra:2012]. 

[^14]: <http://www.cemeai.icmc.usp.br/Euler/index.html>

[^15]: https://github.com/rgmantovani/MfeatExtractor

[^16]: These performance values are assessed by using a nested- resampling method. 

[^17]: imbalance rate = (majority class size / minority class size)

[^18]: This classifier simply predicts the majority class. 

[^19]: The performance values were assessed using the implementations provided by the `mlr` R package. 

[^20]: <http://weka.sourceforge.net/doc.dev/weka/classifiers/trees/J48.html>

[^21]: <https://cran.r-project.org/web/packages/RWeka/index.html> package. 

    The meta-datasets for the J48 algorithm were generated based on tuning results obtained from $102$ datasets reported in [@Mantovani:2016]

[^22]: The J48 hyperparameter space is also presented in \[app:j48\_space\]. 

[^23]: In this paper, the “*hyperparameter profile*” term refers to how sensitive an algorithm may be to the tuning task. 

[^24]: The performance of 1-NN algorithm.  See tables \[tab:mfeats\_1\] and \[tab:mfeats\_2\] in Appendix \[app:features\]. 

[^25]: This is shown in Figure \[fig:rf-mfeat\] in Subsection \[subsec:rf\].