dataesr/scientific-paragraphs-categorization huggingface dataset:
Introduction Until about ten years ago, machine learning (ML) was a discipline little known to the public. For ML scientists, it was a "sellers market": they were producing hosts of algorithms in search for applications and were constantly looking for new interesting datasets. Large internet corporations accumulating massive amounts of data such as Google, Facebook, Microsoft and Amazon have popularized the use of ML and data science competitions have engaged a new generation of young scientists in this wake. Nowadays, government and corporations keep identifying new applications of ML and with the increased availability of open data, we have switched to a "buyers market": everyone seems to be in need of a learning machine. Unfortunately however, learning machines are not yet fully automatic: it is still difficult to figure out which software applies to which problem, how to horseshoe-fit data into a software and how to select properly (hyper-)parameters.
The ambition of the ChaLearn AutoML challenge series is to channel the energy of the ML community to reduce step by step the need for human intervention in applying ML to a wide variety of practical problems. Full automation is an unbounded problem since there can always be novel settings, which have never been encountered before. Our first challenges AutoML1 were limited to: • Supervised learning problems (classification and regression). • Feature vector representations. • Homogeneous datasets (same distribution in the training, validation, and test set). • Medium size datasets of less than 200 MBytes. • Limited computer resources with execution times of less than 20 minutes per dataset on an 8 core x86 64 machine with 56 GB RAM. By doing that, we excluded unsupervised learning, active learning, transfer learning, and causal discovery problems, which are all very dear to us and have been addressed in past ChaLearn challenges  (Guyon, 2011 (Guyon, -2016)) , but which require each a different evaluation setting, thus making result comparisons very difficult.
We did not exclude the treatment of video, images, text, and more generally time series and the selected datasets actually contain several instances of such modalities. However, they were first preprocessed in a feature representation, thus de-emphasizing feature learning. Still, learning from data pre-processed in feature-based representations already covers a lot of grounds and a fully automated method resolving this restricted problem would already be a major advance in the field. Within this constrained setting, we included a variety of difficulties: • Different data distributions: the intrinsic/geometrical complexity of the dataset. • Different tasks: regression, binary classification, multi-class classification, multilabel classification. • Different scoring metrics: AUC, BAC, MSE, F 1 , etc.
This challenge series started with the NIPS 2006 "model selection game" 2   (Guyon et al. , 2011) , where the participants were provided with a machine learning toolbox based on the Matlab toolkit CLOP (Alamdari and  Guyon, 2006)  built on top of "the Spider" package  (Weston et al. , 2007). The toolkit provided a flexible way of building models by combining preprocessing, feature selection, classification and post-processing modules, also enabling the building of ensembles of classifiers. The goal of the game was to build the best hyper-model: the focus was on model selection, not on the development of new algorithms. All problems were feature-based binary classification problems. Five datasets were provided. The participants had to submit the schema of their model. The model selection game confirmed the effectiveness of cross-validation (the winner invented a new variant called cross-indexing) and pre-figured the need to focus more on search effectiveness with the deployment by novel search techniques such as particle swarm optimization.
New in the 2015/2016 AutoML challenge, we introduced the notion of "task": each dataset was supplied with a particular scoring metric to be optimized and a time budget. We initially intended to vary widely the time budget from dataset to dataset in an arbitrary way. We ended up fixing it to 20 minutes for practical reasons (Except for Round 0 where the time budget ranged from 100 to 300 seconds). However, because the datasets varied in size, this put pressure on the participants to manage their allotted time. Other elements of novelty included the freedom of submitting any Linux executable. This was made possible by using automatic execution on the open-source platform Codalab 3. To help the participants we provided a starting kit in Python based on the scikit-learn library  (Pedregosa et al.
, 2011)   4. This induced many of them to write a wrapper around scikit-learn. This has been the strategy of the winning entry "auto-sklearn"  (Feurer et al. , 2015a (Feurer et al. , ,b,c, 2018) )   5. Following the AutoML challenge, we organized a "beat auto-sklearn" game on a single dataset (madeline), in which the participants could provide hyper-parameters "by hand" to try to beat autosklearn. But nobody could beat auto-sklearn! Not even their designers. The participants could submit a json file which describes a sklearn model and hyper-parameter settings, via a GUI interface. This interface usefully allows researchers who want to compare their search methods with auto-sklearn to use the exact same set of hyper-models. A large number of satellite events including bootcamps, summer schools, and workshops have been organized in 2015/2016 around the AutoML challenge.
6 The AutoML challenge was part of the official selection of the competition program of IJCNN 2015 and 2016 and the results were discussed at the AutoML and CiML workshops at ICML and NIPS in 2015 and 2016. Several publications accompanied these events: in  (Guyon et al. , 2015b)  we describe the details of the design of the AutoML challenge 7. In  (Guyon et al. , 2015a)  and  (Guyon et al. , 2016)  we review milestone and final results presented at the ICML 2015 and 2016 AutoML workshops. The 2015/2016 AutoML challenge had 6 rounds introducing 5 datasets each. We also organized a follow-up event for the PAKDD conference 2018 8  in only 2 phases, with 5 datasets in the development phase and 5 datasets in the final "blind test" round. Going beyond the former published analyses, this paper presents systematic studies of the winning solutions on all the datasets of the challenge and conducts comparisons with commonly used learning machines implemented in scikit-learn.
Scope of the Problem This challenge series focuses on supervised learning in ML and, in particular, solving classification and regression problems, without any further human intervention, within given constraints. To this end, we released a large number of datasets pre-formatted in given feature representations (i. e. , each example consists of a fixed number of numerical coefficients; more in Section 3). The distinction between input and output variables is not always made in ML applications. For instance, in recommender systems, the problem is often stated as making predictions of missing values for every variable rather than predicting the values of a particular variable  (Ricci et al. , 2011). In unsupervised learning  (Ghahramani, 2004) , the purpose is to explain data in a simple and compact way, eventually involving inferred latent variables (e.
g. , class membership produced by a clustering algorithm). In the following, only the strict supervised learning setting where data present themselves as identically and independently distributed input-output pairs, is considered. The models used are limited to fixed-length vectorial representations, excluding problems of time series prediction. Text, speech, and video processing tasks included in the challenge have been preprocessed in suitable fixed-length vectorial representations. The difficulty of the proposed tasks lies on the data complexity (class imbalance, sparsity, missing values, categorical variables). The testbed is composed of data from a wide variety of domains. Although there exist ML toolkits that can tackle all these problems, it still requires considerable human effort to find, for a given dataset, task, evaluation metric, the methods and hyper-parameter settings that maximize performance subject to a computational constraint.
, 2013). However, even these methods have many modeling choices and hyper-parameters. While producing models for a diverse range of applications has been a focus of the ML community, little effort has been devoted to the optimization of hyper-parameters. Common practices that include trial and error and grid search may lead to overfitting data for small datasets or underfitting data for large datasets. By overfitting we mean producing models that perform well on training data but perform poorly on unseen data, i. e. , models that do not generalize. By underfitting we mean selecting too simple a model, which does not capture the complexity of the data, and hence performs poorly both on training and test data. Despite well-optimized off-the-shelf algorithms for optimizing parameters, endusers are still responsible for organizing their numerical experiments to identify the best of a number of models under consideration.
Strategies of Model Search Most practitioners use heuristics such as grid search or uniform sampling to sample θ space, and use k-fold cross-validation as the upper-level objective J 2  (Dietterich, 1998). In this framework, the optimization of θ is not performed sequentially  (Bergstra and Bengio, 2012). All the parameters are sampled along a regular scheme, usually in linear or log scale. This leads to a number of possibilities that exponentially increases with the dimension of θ. k-fold cross-validation consists of splitting the dataset into k folds; (k -1) folds are used for training and the remaining fold is used for testing; eventually, the average of the test scores obtained on the k folds is reported. Note that some ML toolkits currently support cross-validation. There is a lack of principled guidelines to determine the number of grid points and the value of k (with the exception of Dietterich (1998)), and there is no guidance for regularizing J 2 , yet this simple method is a good baseline approach.
Bayesian methods introduce techniques of overfitting avoidance via the notion of hyper-priors, but at the expense of making assumptions on how the data were generated and without providing guarantees of performance. In all the prior approaches to full model selection we know of, there is no attempt to treat the problem as the optimization of a regularized functional J 2 with respect to both (1) modeling choices and (2) data split. Much remains to be done to jointly address statistical and computational issues. The AutoML challenge series offers benchmarks to compare and contrast methods addressing these problems, free of the inventor/evaluator bias.
Data We gathered a first pool of 70 datasets during the summer 2014 with the help of numerous collaborators and ended up selecting 30 datasets for the 2015/2016 challenge (see Table  2  and Appendix B), chosen to illustrate a wide variety of domains of applications: biology and medicine, ecology, energy and sustainability management, image, text, audio, speech, video and other sensor data processing, Internet social media management and advertising, market analysis and financial prediction. We preprocessed data to obtain feature representations   (i. e. , each example consists of a fixed number of numerical coefficients). Text, speech, and video processing tasks were included in the challenge, but not in their native variable length representations. For the 2018 challenge, three datasets from the first pool (but unused in the first challenge) were selected and 7 new datasets collected by the new organizers and sponsors were added (see Table  3  and Appendix C ).
Some datasets were obtained from public sources, but they were reformatted into new representations to conceal their identity, except for the final round of the 2015/2016 challenge and the final phase of the 2018 challenge, which included completely new data. In the 2015/2016 challenge, data difficulty progressively increased from round to round. Round 0 introduced five (public) datasets from previous challenges illustrating the various difficulties encountered in subsequent rounds: Novice. Binary classification problems only. No missing data; no categorical features; moderate number of features (< 2, 000); balanced classes. Challenge lies in dealing with sparse and full matrices, presence of irrelevant variables, and various P tr/N. Intermediate. Binary and multi-class classification problems.
Challenge lies in dealing with unbalanced classes, number of classes, missing values, categorical variables, and up to 7,000 features. Advanced. Binary, multi-class, and multi-label classification problems. Challenge lies in dealing with up to 300,000 features. Expert. Classification and regression problems. Challenge lies in dealing with the entire range of data complexity. Master. Classification and regression problems of all difficulties. Challenge lies in learning from completely new datasets. The datasets of the 2018 challenge were all binary classification problems. Validation partitions were not used because of the design of this challenge, even when they were available for some tasks. The three reused datasets had similar difficulty as those of rounds 1 and 2 of the 2015/2016 challenge.
Challenge Protocol In this section, we describe design choices we made to ensure the thoroughness and fairness of the evaluation. As previously indicated, we focus on supervised learning tasks (classification and regression problems), without any human intervention, within given time and computer resource constraints (Section 4. 1), and given a particular metric (Section 4. 2), which varies from dataset to dataset. During the challenges, the identity and description of the datasets is concealed (except in the very first round or phase where sample data is distributed) to avoid the use of domain knowledge and to push participants to design fully automated ML solutions. In the 2015/2016 AutoML challenge, the datasets were introduced in a series of rounds (Section 4. 3), alternating periods of code development (Tweakathon phases) and blind tests of code without human intervention (AutoML phases).
Time Budget and computational resources The Codalab platform provides computational resources shared by all participants. We used up to 10 compute workers processing in parallel the queue of submissions made by participants. Each compute worker was equipped with 8 cores x86 64. Memory was increased from 24 GB to 56 GB after round 3 of the 2015/2016 AutoML challenge. For the 2018 AutoML challenge computing resources were reduced, as we wanted to motivate the development of more efficient yet effective AutoML solutions. We used 6 compute workers processing in parallel the queue of submissions. Each compute worker was equipped with 2 cores x86 64 and 8 GB of memory. To ensure fairness, when a code submission was evaluated, a compute worker was dedicated to processing that submission only, and its execution time was limited to a given time budget (which may vary from dataset to dataset).
The time budget was provided to the participants with each dataset in its info file. It was generally set to 1200 seconds (20 minutes) per dataset, for practical reasons, except in the first phase of the first round. However, the participants did not know this ahead of time and therefore their code had to be capable to manage a given time budget. The participants who submitted results -instead of code-were not constrained by the time budget since their code was run on 9. In RL, PM, RH, RI and RM datasets instances were chronologically sorted, this information was made available to participants and could be used for developing their methods. their own platform. This was potentially advantageous for entries counting towards the Final phases (immediately following a Tweakathon). Participants wishing to also enter the AutoML (blind testing) phases, which required submitting code, could submit both results and code (simultaneously).
When results were submitted, they were used as entries in the on-going phase. They did not need to be produced by the submitted code; i. e. if a participant did not want to share personal code, he/she could submit the sample code provided by the organizers together with his/her results. The code was automatically forwarded to the AutoML phases for "blind testing". In AutoML phases, result submission was not possible. The participants were encouraged to save and submit intermediate results so we could draw learning curves. This was not exploited during the challenge. But we study learning curves in this paper to evaluate the capabilities of algorithms to quickly attain good performances.
Each dataset has its own scoring criterion specified in its info file. All scores are normalized such that the expected value of the score for a random prediction, based on class prior probabilities, is 0 and the optimal score is 1. Multi-label problems are treated as multiple binary classification problems and are evaluated using the average of the scores of each binary classification subproblem. We first define the notation • for the average over all samples P indexed by i. That is, y i = (1/P ) P i=1 (y i ). (2) The score metrics are defined as follows: R 2. The coefficient of determination is used for regression problems only. The metric is based on the mean squared error (MSE) and the variance (VAR), and computed as R 2 = 1 -MSE /VAR, (3) where MSE = (y i -q i ) 2 and VAR = (y i -m) 2 , with m = y i.
Rounds and Phases in the 2015/2016 challenge The 2015/2016 challenge was run in multiple phases grouped in six rounds. Round 0 (Preparation) was a practice round using publicly available datasets. It was followed by five rounds of progressive difficulty (Novice, Intermediate, Advanced, Expert, and Master). Except for rounds 0 and 5, all rounds included three phases that alternated AutoML and Tweakathons contests. These phases are described in Table  4. Submissions were made in Tweakathon phases only. The results of the latest submission were shown on the leaderboard and such submission automatically migrated to the following phase. In this way, the code of participants who abandoned before the end of the challenge had a chance to be tested in subsequent rounds and phases. New participants could enter at any time.
Prizes were awarded in phases marked with a * during which there was no submission. To participate in phase AutoML  [n] , code had to be submitted in Tweakathon  [n-1]. In order to encourage participants to try GPUs and deep learning, a GPU track sponsored by NVIDIA was included in Round 4. To participate in the Final[n], code or results had to be submitted in Tweakathon  [n]. If both code and (well-formatted) results were submitted, the results were used for scoring rather than rerunning the code in Tweakathon[n] and Final  [n]. The code was executed when results were unavailable or not well formatted. Thus, there was no disadvantage in submitting both results and code. If a participant submitted both results and code, different methods could be used to enter the Tweakathon/Final phases and the AutoML phases.
Phases in the 2018 challenge The 2015/2016 AutoML challenge was very long and few teams participated in all rounds. Further, even though there was no obligation to participate in previous rounds to enter new rounds, new potential participants felt they would be at a disadvantage. Hence, we believe it is preferable to organize recurrent yearly events, each with their own workshop and publication opportunity. This provides a good balance between competition and collaboration. In 2018, we organized a single round of AutoML competition in two phases. In this simplified protocol, the participants could practice on five datasets during the first (development) phase, by either submitting code or results. Their performances were revealed immediately, as they became available, on the leaderboard. The last submission of the development phase was automatically forwarded to the second phase: the AutoML "blind test" phase.
In this second phase, which was the only one counting towards the prizes, the participants' code was automatically evaluated on five new datasets on the Codalab platform. The datasets were not revealed to the participants. Hence, submissions that did not include code capable of being trained and tested automatically were not ranked in the final phase and could not compete towards the prizes. We provided the same starting kit as in the AutoML 2015/2016 challenge, but the participants also had access to the code of the winners of the previous challenge.
AutoML Final Rnd Ended Winners < R > < S > Ended Winners < R > < S > UP (%) 1. ideal 1. There is still room for improvement, as revealed by the significant differences remaining between Tweakathon and AutoML (blind testing) results (Table  5  and Figure   3-b ). In Round 3, all but one participant failed to turn in working solutions during blind testing, because of the introduction of sparse datasets. Fortunately, the participants recovered, and, by the end of the challenge, several submissions were capable of returning solutions on all the datasets of the challenge. But learning schemas can still be optimized because, even discarding Round 3, there is a 15 to 35% performance gap between AutoML phases (blind testing with computational constraints) and Tweakathon phases (human intervention and additional compute power).
Scores obtained in the 2018 challenge The 2018 challenge lasted 4 months  (Nov. 30, 2017 to March 31, 2018). As in the previous challenge, top ranked solutions were obtained and open sourced. Table  6  shows the results of both phases of the 2018 challenge. As a reminder, this challenge had a feedback phase and a blind test phase, the performances of the winners in each phase are reported. Performance in this challenge was slightly lower than that observed in the previous edition. This was due to the difficulty of the tasks (see below) and the fact that data sets in the feedback phase included three deceiving datasets (associated to tasks from previous challenges, but not necessarily similar to the data sets used in the blind test phase) out of five. We decided to proceed this way to emulate a realistic AutoML setting.
Difficulty of datasets/tasks In this section, we assess dataset difficulty, or rather task difficulty since the participants had to solve prediction problems for given datasets, performance metrics, and computational time constraints. The tasks of the challenge presented a variety of difficulties, but those were not equally represented (Tables  2, 3 ): • Categorical variables and missing data: Few datasets had categorical variables in the 2015/2016 challenge (ADULT, ALBERT, and WALDO), and not very many variables were categorical in those datasets. Likewise, very few datasets had missing values (ADULT and ALBERT) and those included only a few missing values. So neither categorical variables nor missing data presented a real difficulty in this challenge, though ALBERT turned out to be one of the most difficult datasets because it was also one of the largest ones.
This situation changed drastically for the 2018 challenge where five out of the ten datasets included categorical variables (RL, PM, RI, RH and RM) and missing values (GINA, PM, RL, RI and RM). These were among the main aspects that caused the low performance of most methods in the blind test phase. • Large number of classes. Only one dataset had a large number of classes (DIONIS with 355 classes). This dataset turned out to be difficult for participants, particularly because it is also large and has unbalanced classes. However, datasets with large number of classes are not well represented in this challenge. HELENA, which has the second largest number of classes (100 classes), did not stand out as a particularly difficult dataset. However, in general, multi-class problems were found more difficult than binary classification problems.