dataesr/scientific-paragraphs-categorization huggingface dataset:
• Regression. We had only four regression problems: CADATA, FLORA, YOLANDA, PABLO. Figure  7 : Meta-features most predictive of dataset intrinsic difficulty (2015/2016 challenge data). Meta-feature GINI importances are computed by a random forest regressor, trained to predict the highest participant leaderboard score using meta-features of datasets. Description of these meta-features can be found in Table  1  of the supplementary material of  Feurer et al. (2015b). Blue and red colors respectively correspond to positive and negative correlations (Pearson correlations between meta features and score medians). • Sparse data. A significant number of datasets had sparse data (DOROTHEA, FABERT, ALEXIS, WALLIS, GRIGORIS, EVITA, FLORA, TANIA, ARTURO, MARCO). Several of them turned out to be difficult, particularly ALEXIS, WALLIS, and GRIG-ORIS, which are large datasets in sparse format, which cause memory problems when they were introduced in round 3 of the 2015/2016 challenge.
We later increased the amount of memory on the servers and similar datasets introduced in later phases caused less difficulty. • Large datasets. We expected the ratio of the number N of features over the number P tr of training examples to be a particular difficulty (because of the risk of overfitting), but modern machine learning algorithm are robust against overfitting. The main difficulty was rather the PRODUCT N * P tr. Most participants attempted to load the entire dataset in memory and convert sparse matrices into full matrices. This took very long and then caused loss in performances or program failures. Large datasets with N * P tr > 20. 10 6 include ALBERT, ALEXIS, DIONIS, GRIGORIS, WAL-LIS, EVITA, FLORA, TANIA, MARCO, GINA, GUILLERMO, PM, RH, RI, RICCARDO, RM. Those overlap significantly with the datasets with sparse data (in bold).
For the 2018 challenge, all data sets in the final phase exceeded this threshold, and this was the reason of why the code from several teams failed to finish within the time budget. Only ALBERT and DIONIS were "truly" large (few features, but over 400,000 training examples). • Presence of probes: Some datasets had a certain proportion of distractor features or irrelevant variables (probes). Those were obtained by randomly permuting the values of real features. Two-third of the datasets contained probes ADULT, CADATA, DIG-ITS, DOROTHEA, CHRISTINE, JASMINE, MADELINE, PHILIPPINE, SYLVINE, ALBERT, DILBERT, FABERT, JANNIS, EVITA, FLORA, YOLANDA, ARTURO, CARLO, PABLO, WALDO. This allowed us in part to make datasets that were in the public domain less recognizable. • Type of metric: we used 6 metrics, as defined in section 4.
2. The distribution of tasks in which they were used was not uniform: BAC (11), AUC (6), F1 (3), and PAC (6) for classification, and R2 (2) and ABS (2) for regression. This is because not all metrics lend themselves naturally to all types of applications. • Time budget: Although in round 0 we experimented with giving different time budgets for the various datasets, we ended up assigning 1200 seconds (20 min) to all datasets in all other rounds. Because the datasets varied in size, this put more constraints on large datasets. • Class imbalance: This was not a difficulty found in the 2015/2016 datasets. However, extreme class imbalance was the main difficulty for the 2018 edition. Imbalance ratios lower or equal to 1 to 10 were present in RL, PM, RH, RI, and RM datasets, in the latter data set class imbalance was as extreme as 1 to 1000.
This was the reason why the performance of teams was low. Figure  4  gives a first view of dataset/task difficulty for the 2015/2016 challenge. It captures, in a schematic way, the distribution of the participants' performance in all rounds on test data, in both AutoML and Tweakathon phases. One can see that the median performance over all datasets improves between AutoML and Tweakathon, as can be expected. Correspondingly, the average spread in performance (quartile) decreases. Let us take a closer look at the AutoML phases: The "accident" of round 3 in which many methods failed in blind testing is visible (introduction of sparse matrices and larger datasets) 10. Round 2 (multi-class classification) appears to have also introduced a significantly higher degree of difficulty than round 1 (binary classification).
In round 4, two regression problems were introduced (FLORA and YOLANDA), but it does not seem that regression was found significantly harder than multiclass classification. In round 5 no novelty was introduced. We can observe that, after round 3, the dataset median scores are scattered around the overall median. Looking at the corresponding scores in the Tweakathon phases, one can remark that, once the participants recovered from their surprise, round 3 was not particularly difficult for them. Rounds 2 and 4 were comparatively more difficult. For the datasets used in the 2018 challenge, the tasks difficulty was clearly associated to extreme class imbalance, inclusion of categorical variables and high dimensionality in 10. Examples of sparse datasets were provided in round 0, but they were of smaller size.
terms of N × P tr. However, for the 2015/2016 challenge data sets we found that is was generally difficult to guess what makes a task easy or hard, except for dataset size, which pushed participants to the frontier of the hardware capabilities and forced them to improve the computational efficacy of their methods. Binary classification problems (and multilabel problems) are intrinsically "easier" than multiclass problems, for which "guessing" has a lower probablity of success. This partially explains the higher median performance in rounds 1 and 3, which are dominated by binary and multi-label classification problems. There is not a large enough number of datasets illustrating each type of other difficulties to draw other conclusions. We ventured however to try to find summary statistics capturing overall taks difficulty.
Figures  5  and 6  give representations of our estimates of intrinsic and modeling difficulties for the 2015/2016 challenge datasets. It can be seen that the datasets of round 0 were among the easiest (except perhaps NEWSGROUP). Those were relatively small (and well-known) datasets. Surprisingly, the datasets of round 3 were also rather easy, despite the fact that most participants failed on them when they were introduced (largely because of memory limitations: scikit-learn algorithms were not optimized for sparse datasets and it was not possible to fit in memory the data matrix converted to a dense matrix). Two datasets have a small intrinsic difficulty but a large modeling difficulty: MADELINE and DILBERT. MADELINE is an artificial dataset very non-linear (clusters or 2 classes positionned on the vertices of a hyper-cube in a 5 dimensional space) so very hard for Naive Bayes and DILBERT is an image recognition dataset with images of objects rotated in all sorts of positions, also very hard for Naive Bayes.
The datasets of the last 2 phases seem to have a large intrinsic difficulty compared to the modeling difficulty. But this can be deceiving because the datasets are new to the machine learning community and the performances of the winners may still be far from the best attainable performance. We attempted to predict the intrinsic difficulty (as measured by the winners' performance) from the set of meta features used by AAD Freiburg for meta-learning  (Feurer et al. , 2015b) , which are part of OpenML  (Vanschoren et al. , 2014) , using a Random Forest classifier and ranked the meta features in order of importance (most selected by RF). The list of meta features is provided in Appendix A. The three meta-features that predict best dataset difficulty (Figure  7 ) are: • LandmarkDecisionTree: performance of the decision tree classifier.
Hyper-parameter optimization Many participants used the scikit-learn (sklearn) package, including the winning group AAD Freiburg, which produced the auto-sklearn software. We used the auto-sklearn API to conduct post-challenge systematic studies of the effectiveness of hyper-parameter optimization. We compared the performances obtained with default hyper-parameter settings in scikit-learn and with hyper-parameters optimized with auto-sklearn 12 , both within the time budgets as imposed during the challenge, for four "representative" basic methods: k-nearest neighbors (KNN), naive Bayes (NB), Random Forest (RF), and a linear model trained with stochastic gradient descent (SGD-linear 13 ). The results are shown in Figure  8. We see that hyper-parameter optimization usually improves performance, but not always.
The advantage of hyperparameter tuning comes mostly from its flexibility of switching the optimization metric to the one imposed by the task and from finding hyperparameters that work well given the current dataset and metric. However, in some cases it was not possible 12. we use sklearn 0. 16. 1 to mimic the challenge environment, and auto-sklearn 0. 4. 0. 13. we set the loss of SGD to be 'log' in scikit-learn for these experiments to perform hyperparameter optimization within the time budget due to the data set size (score ≤ 0). Thus, there remains future work on how to perform thorough hyperparameter tuning given rigid time constraints and huge datasets. We also compared the performances obtained with different scoring metrics (Figure  9 ). Basic methods do not give a choice of metrics to be optimized, but auto-sklearn postfitted the metrics of the challenge tasks.
Meta-learning One question is whether meta-learning  (Brazdil et al. , 2008)  is possible, that is learning to predict whether a given classifier will perform well on future datasets (without actually training it), based on its past performances on other datasets. We investigated whether it is possible to predict which basic method will perform best based on the meta-learning features of auto-sklearn (see Appendix A). We removed the "Landmark" features from the set of meta features because those are performances of basic predictors (albeit rather poor ones with many missing values), which would lead to a form of "data leakage". We used four basic predictors: • NB: Naive Bayes • SGD-linear: Linear model (trained with stochastic gradient descent) • KNN: K-nearest neighbors • RF: Random Forest We used the implementation of the scikit-learn library and with default hyper-parameter settings.
While a few years ago the hottest issue in model selection and hyper-parameter optimization was over-fitting, in present days the problem seems to have been largely avoided by using ensembling techniques. In the 2015/2016 challenge, we varied the ratio of number of training examples over number of variables (P tr/N ) by several orders of magnitude. Five datasets had a ratio P tr/N lower than one (dorothea, newsgroup, grigoris, wallis, and flora), which is a case lending itself particularly to overfitting. Although P tr/N is the most predictive variable of the median performance of the participants, there is no indication that the datasets with P tr/N < 1 were particularly difficult to the participants (Figure  5 ). Ensembles of predictors have the additional benefit of addressing in a simple way the "any-time learning" problem by growing progressively a bigger ensemble of predictors, improving performance over time.
Search strategies: Filter, wrapper, and embedded methods With the availability of powerful machine learning toolkits like scikit-learn (on which the starting kit was based), the temptation is great to implement all-wrapper methods to solve the CASH (or "full model selection") problem. Indeed, most participants went that route. Although a number of ways of optimizing hyper-parameters with embedded methods for several basic classifiers have been published  (Guyon et al. , 2006a) , they each require changing the implementation of the basic methods, which is time consuming and error prone compared to using already debugged and well optimized library version of the methods. Hence practitioners are reluctant to invest development time in the implementation of embedded methods. A notable exception is the software of marc.
To compare strategies, we show in Appendix G learning curves for two top ranking participants who adopted very different methods: Abhishek and aad freiburg. The former uses heuristic methods based on prior human experience while the latter initializes search with models predicted to be best suited by a meta-learning machine, then performs Bayesian optimization of hyper-parameters. Abhishek seems to often start with a better solution but performs a less efficient exploration. In contrast, aad freiburg starts lower but often ends up with a better solution. Some elements of randomness in the search are useful to arrive at better solutions. Preprocessing and feature selection: The datasets presented themselves with intrinsic difficulties that could be in part addressed by preprocessing or special modifications of algorithms: Sparsity, missing values, categorical variables, and irrelevant variables.
See Appendix D for more details. Transfer learning and meta learning: To our knowledge, only the aad freiburg relied on meta-learning to initialize their hyper-parameter search. To that end, they used datasets of OpenML 14. The number of datasets released and the diversity of tasks did not allow the participants to perform effective transfer learning or meta learning. Deep learning: The type of computations resources available in AutoML phases ruled out the use of Deep Learning, except in the GPU track. However, even in that track, the Deep Learning methods did not come ahead. One exception is the aad freiburg, they used Deep Learning in Tweakathon rounds three and four and found it to be helpful for the datasets Alexis, Tania and Yolanda. Task and metric optimization: There were four types of tasks (regression, binary classification, multi-class classification, and multi-label classification) and six scoring metrics (R2, ABS, BAC, AUC, F1, and PAC).
Moreover, class balance and number of classes varied a lot for classification problems. Moderate effort has been put into designing methods optimizing specific metrics. Rather, generic methods were used and the outputs post-fitted to the target metrics by cross-validation or through the ensembling method. Engineering: One of the big lessons of the AutoML challenge series is that most methods fail to return results in all cases, not a "good" result, but "any" reasonnable result. Reasons for failure include "out of time" and "out of memory" or various other failures (e. g. numerical instabilities). We are still very far from having "basic models" that run on all datasets. One of the strength of auto-sklearn is to ignore those models that fail and generally find at least one that returns a result.
Discussion We briefly summarize the main questions we asked ourselves and the main findings: 1. Was the provided time budget sufficient to complete the tasks of the challenge? We drew learning curves as a function of time for the winning solution of aad f reiburg (auto-sklearn), see Appendix G). This revealed that for most dataset performances still improved well beyond the time limit imposed by the organizers. Although for about half the datasets the improvement is modest (no more that 20% of the score obtained at the end of the imposed time limit), for some datasets the improvement was very large (more than 2x the original score). The improvements are usually gradual, but sudden performance improvements occur. For instance, for Wallis, the score doubled suddenly at 3x the time limit imposed in the challenge.
As also noted by the authors of the package  (Feurer et al. , 2015b)  auto-sklearn has a slow start but on the long run gets performances close to the best method. 2. Are there tasks that were significantly more difficult than others for the participants? Yes, there was a very wide range of difficulties for the tasks as revealed by the dispersion of the participants in terms of average (median) and variability (third quartile) of their scores. Madeline, a synthetic dataset featuring a very non-linear task, was very difficult for many participants. Other difficulties that caused failures to return a solution included large memory requirements (particularly for methods that attempted to convert sparse matrices to full matrices), and short time budget for datasets with large number of training examples and/or features or with many classes or labels.
3. Are there meta-features of datasets and methods providing useful insight to recommend certain methods for certain types of datasets? The aad freiburg team used a subset of 53 meta-features (a superset of the simple statistics provided with the challenge datasets) to measure similarity between datasets. This allowed them to conduct hyper-parameter search more effectively by initializing the search with settings identical to those selected for similar datasets previously processed (a form of meta-learning). Our own analysis revealed that it is very difficult to predict the predictors' performances from the meta-features, but it is possible to predict relatively accurately which "basic method" will perform best. With LDA, we could visualize how datasets recoup in two dimensions and show a clean separation between datasets "preferring" Naive Bayes, linear SGD, or KNN, or RF.
Conclusion We have analyzed the results of several rounds of AutoML challenges. Our design of the first AutoML challenge (2015/2016) was satisfactory in many respects. In particular, we attracted a large number of participants (over 600), attained results that are statistically significant, and advanced the state-of-the-art to automate machine learning. Publicly available libraries have emerged as a result of this endeavor, including "autosklearn". In particular, we designed a benchmark with a large number of diverse datasets, with large enough test sets to separate top ranking participants. It is difficult to anticipate the size of the test sets needed, because the error bars depend on the performances attained by the participants, so we are pleased that we made reasonable guesses. Our simple rule-ofthumb "N=50/E" where N is the number of test samples and E the error rate of the most depleted class seems to be widely applicable.
Also concerning the datasets, we made sure that they were neither too easy nor too hard. This is important to be able to separate participants. To quantify this, we introduced the notion of "intrinsic difficulty" and "modeling difficulty". Intrinsic difficulty can be quantified by the performance of the best method (as a surrogate for the best attainable performance, i. e. the Bayes rate for classification problems). Modeling difficulty can be quantified by the spread in performance between methods. Our best problems have relatively low "intrinsic difficulty" and high "modeling difficulty". However, the diversity of the 30 datasets of our first 2015/2016 challenge is both a feature and a curse: it allows us to test the robustness of software across a variety of situations, but is makes meta-learning very difficult, if not impossible (datasets being very little redundant with respect to meta-features).
Consequently, external meta-learning data must be used if meta-learning is to be explored. This was the strategy adopted by the AAD Freiburg team, which used the OpenML data for meta training. Likewise, we attached different metrics to each dataset. This contributed to making the tasks more realistic and more difficult, but also made meta-learning harder. In the second 2018 challenge, we diminished the variety of datasets and used a single metric. With respect to task design, we learned that the devil is in the details. The challenge participants solve exactly the task proposed to the point that their solution may not be adaptable to seemingly similar scenarios. In the case of the AutoML challenge, we pondered whether the metric of the challenge should be the area under the learning curve or one point on the learning curve (the performance obtained after a fixed maximum computational time elapsed).
g. people working remotely via an unreliable connection). This will influence the design of future challenges. Also regarding task design, the two versions of AutoML challenge we have run differ in the difficulty of transfer learning. In the 2015/2016 challenge, round 0 introduced a sample of all types of data and difficulties (types of targets, sparse data or not, missing data or not, categorical variables of not, more examples than features or not). Then each round ramped up difficulty. But in fact the datasets of round 0 were relatively easy. Then at each round, the code of the participants was blind tested on data that were one notch harder than in the previous round. Hence transfer was quite hard. In the 2018 challenge, we had 2 phases, each with 5 datasets of similar difficulty and the datasets of the first phase were each matched with one corresponding dataset on a similar task.
• train data aspect ratio = ratio of number of training examples over number of features We define the meta features as implemented in  (Feurer et al. , 2015a,b,c)   • Landmark[Some Model]: accuracy of [Some Model] applied on dataset. • LandmarkDecisionNodeLearner & LandmarkRandomNodeLearner: Both are decision tree with max depth=1. 'DecisionNode' considers all features when looking for best split, and 'RandomNode' considers only 1 feature, where comes the term 'random'. • Skewnesses: Skewness of each numerical features. Skewness measures the symmetry of a distribution. A skewness value > 0 means that there is more weight in the left tail of the distribution. Computed by scipy. stats. skew. • SkewnessMax / SkewnessMin / SkewnessMean / SkewnessSTD: max / min / mean / std over skewness of all features.
• NumberOfMissingValues: Total number of missing value Description: This is a dataset of handwritten digits. It is a subset of a larger set made available from NIST. The digits in pixel representation have been size-normalized and centered in a fixed-size image by the authors. The data are quantized on 256 gray level values. Preparation: For the purpose of the AutoML challenge, all samples were merged and the data were freshly randomly split in three sets: training, validation, and test. The order of the features (pixels) was also randomize, after adding a few distractor features (probes) that are permuted versions of real features. Representation: Pixels. SET 0. 4: DOROTHEA Description: Synopsis of the original data: One binary attribute (active A or inactive I) must be predicted. Drugs are typically small organic molecules that achieve their desired activity by binding to a target site on a receptor.
The order of the patterns and the order of the features were globally randomly permuted to mix the original training and the test patterns and remove the feature order. The data was split into training, validation, and test set while respecting the same proportion of examples of the positive and negative class in each set. SET 0. 5: NEWSGROUPS This dataset was prepared by Hugo Jair Escalante from original data provided by Ken Lang. The version we used was obtained from Deng Cai. : "TNW -20 Newsgroups data set" (http://www. cad. zju. edu. cn/home/dengcai/Data/TextData. html). Past Usage: The 20 NewsGroups data set is among the most used data sets for text categorization. It has been used to evaluate standard text categorization and recently it has been also widely used for the evaluation of cross domain text categorization.