dataesr/scientific-paragraphs-categorization huggingface dataset:
Description: In this version of the data set the training and test documents were mixed in a single matrix, then plit into training, validation, and test set for the needs of the challenge. Preparation: The data is organized into 20 different newsgroups (each newsgroup corresponds to a class), each corresponding to a different topic (see http://qwone. com/ ~jason/20Newsgroups/). Some of the newsgroups are very closely related to each other (e. g. comp. sys. ibm. pc. hardware / comp. sys. mac. hardware), while others are highly unrelated (e. g misc. forsale / soc. religion. christian). Representation: Documents are represented by their bag-of-words using a termfrequency weighting scheme. This dataset was prepared by Isabelle Guyon from original data provided by Curt Breneman, Charles Bergeron, and Kristin Bennett: "Activation of pyruvate kynase" (http: //www.
causality. inf. ethz. ch/activelearning. php?page=datasets). Past Usage: Active learning challenge, C dataset, see http://www. causality. inf. ethz. ch/activelearning. php. Description: The task is to predict chemical activity of molecules. This is a two-class classification problems. The variables represent properties of the molecule inferred from its structure. The problem is therefore to relate structure to activity (a QSAR=quantitative structure-activity relationship problem) to screen new compounds before actually testing them (a HTS=high-throughput screening problem). The problem is to predict the activation of pyruvate kynase, a well characterized enzyme, which regenerates ATP in glycolysis by catalyzing phosphoryl transfer from phosphoenol pyruvate to ADP to yield pyruvate and ATP.
Preparation: We modified the original data split and added probes. Representation: Features/Attributes representing properties of molecules. SET 1. 2: JASMINE This dataset was prepared by Isabelle Guyon from original data provided by Isabelle Guypn: "Feature selection challenge data" (http://www. nipsfsc. ecs. soton. ac. uk/datasets/ ;https://archive. ics. uci. edu/ml/datasets/Madelon). Past Usage: NIPS 2003 feature selection challenge. See Result analysis of the NIPS 2003 feature selection challenge, Isabelle Guyon, Steve R. Gunn, Asa Ben-Hur, Gideon Dror, 2004. Description: MADELON is an artificial dataset containing data points grouped in 32 clusters placed on the vertices of a five dimensional hypercube and randomly labeled +1 or -1. The five dimensions constitute 5 informative features.
15 linear combinations of those features were added to form a set of 20 (redundant) informative features. Based on those 20 features one must separate the examples into the 2 classes (corresponding to the ±1 labels). We added a number of distractor feature called *probes* having no predictive power. The order of the features and patterns were randomized. See http: //www. nipsfsc. ecs. soton. ac. uk/papers/NIPS2003-Datasets. pdf. Preparation: To draw random data, the program takes the following steps: (1) Each class is composed of a number of Gaussian clusters. N(0,1) is used to draw for each cluster num usef ul f eat examples of independent features. (2) Some covariance is added by multiplying by a random matrix A, with uniformly distributed random numbers between -1 and 1. (3) The clusters are then placed at random on the vertices of a hypercube in a num usef ul f eat dimensional space.
The hypercube vertices are placed at values ±1 class sep. (4) Redundant features are added. They are obtained by multiplying the useful features by a random matrix B, with uniformly distributed random numbers between -1 and 1. (5) Some of the previously drawn features are repeated by drawing randomly from useful and redundant features. Useless features (random probes) are added using N(0,1). (  6 )-All the features are then shifted and rescaled randomly to span 3 orders of magnitude. (7) Random noise is then added to the features according to N(0,0. 1). (  8 ) A fraction f lip y of labels are randomly exchanged. Representation: Continuous valued features. SET 1. 4: PHILIPPINE This dataset was prepared by Hugo Jair Escalante from original data provided by Olivier Chapelle (CRITEO): "Criteos Delayed Feedback in Display Advertising" (https://www.
Variables 1-13 are numeric, variables 14-39 are categorical. Representation: Features related to click prediction, the semantics of the features has not been described elsewhere. This dataset was prepared by Hugo Jair Escalante from original data provided by Fu Jie Huang, Yann LeCun, Leon Bottou: "NORB data set (2 feature maps)" (http://cs. nyu. edu/ ~ylclab/data/norb-v1. 0/). Past Usage: This data set has been widely used for the evaluation of 3D object classifcation, it has been very popular recently for deep learning computer vision, the paper introducing the data set is: Yann LeCun, Fu Jie Huang, Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting, CVPR, 2004. (google scholar reports 380 citations). Description: The data set has 48600 images comprising 5 categories, images come from 50 toys belonging to 5 categories.
SET 2. 3: FABERT We then cut the images into patches of 10x10 pixels and ran kmeans clustering (an on-line version) to create 144 cluster centers. We used these cluster centers as a dictionary to create features corresponding to the presence of one the 144 shapes at one of 25 positions on a grid. This created another 144 x 25 = 3600 features. See http://jmlr. org/proceedings/ papers/v27/supplemental/datasetsutl12a. pdf for details. Representation: Bag of word features. SET 2. 5: VOLKERT This dataset was prepared by Hugo Jair Escalante from original data provided by J. Vogel and B. Schiele. : "VOGEL data set -image classification" (http://ccc. inaoep. mx/ ~hugojair/ebm/ebm_code_and_data. zip). Past Usage: This data set has been used in a few publications for the evaluation of region labeling and image retrieval techniques.
In the past 10 fold CV has been used for evaluation. Representation: Images are represented by their edge and HSI-color histograms, as well as by texture features extracted from the co-occurrence matrix. This dataset was prepared by Isabelle Guyon from original data provided by The dataset was constructed from the KTH human action recognition dataset of Ivan Laptev and Barbara Caputo and the Hollywood 2 dataset of human actions and scenes of Marcin Marszalek, Ivan Laptev, and Cordelia Schmidt. : "Action recognition (from Unsupervised and Transfer Learning Challenge)" (http://www. nada. kth. se/cvap/actions/,http://www. irisa. fr/vista/Equipe/People/Laptev/download. html). Past Usage: The data were used in the Unsupervised and Transfer Learning challenge: http://www. causality. inf. ethz. ch/unsupervised-learning.
php. Description: The data include video clips of people performing actions. The identification and recognition of gestures, postures and human behaviors has gained importance in applications such as video surveillance, gaming, marketing, computer interfaces and interpretation of sign languages for the deaf. Preparation: The data were preprocessed into STIP features using the code of Ivan Laptev: http://www. irisa. fr/vista/Equipe/People/Laptev/download/stip-1. 0-winlinux. zip. The final representation is a ?bag of STIP features?. Details are found in the report http://jmlr. org/proceedings/papers/v27/supplemental/datasetsutl12a. pdf. Representation: Bag of word features. SET 3. 2: DIONIS Description: This is a dataset of Urdu printed ligatures shapes with diacritics stripped off. The dataset has been derived from an original dataset found at : http://www.
cle. org. pk/clestore/imagecorpora. htm by generating new images from existing ones. A subset of shapes is included in this dataset. Preparation: For the purpose of the AutoML challenge, new shape images were created using elastic deformations, rotations, shear and scaling. Features were then extracted from the generated images. Representation: DCT transform of image contours, spatial density computed by dividing each image in a 3x3 grid and computing the density for each cell, eigen values and eigen vector of (x,y) coordinates of foreground shape pixels. Past Usage: The data set is being used in the WISE 2014 -Greek Media Monitoring Multilabel Classification, the challenge is being managed in the Kaggle platform, at the moment of writing this file 121 teams have registered for the competition, these are teams that have made at least one submission.
Each instance is associated to a region of an image. Regions in images have been segmented manually, each region is described by a 27-dimensional verctor comprising the following visual-content attributes: area, boundary/area, width and height of the region, average and standard deviation in x and y, convexity, average, standard deviation and skewness in the RGB and CIE-Lab color spaces. In the past, 10-fold cross validation has been used for evaluation. Preparation: For the purpose of the AutoML challenge, all regions are labeled by the first-level branch of the original labels. Representation: Region area, boundary/area, width and height of the region, average and standard deviation in x and y, convexity, average, standard deviation and skewness in the RGB and CIE-Lab color spaces. This dataset was prepared by Hugo Jair Escalante from original data provided by Hugo Jair Escalante, Michael Grubinger: "SAIAPR TC12 benchmark -top-100 frequent labels" (http://imageclef.
4: TANIA This dataset was prepared by Isabelle Guyon from original data provided by The original data were donated by Reuters and downloaded from: Lewis, D. D. RCV1-v2/LYRL2004: The LYRL2004 Distribution of the  RCV1-v2 Text Categorization Test Collection (12-Apr-2004 Version). : "Text classification (from REUTERS data)" (http://www. jmlr. org/ papers/volume5/lewis04a/lyrl2004_rcv1v2_README). Past Usage: The data were used in the Unsupervised and Transfer Learning challenge: http://www. causality. inf. ethz. ch/unsupervised-learning. php.
Description: We used a subset of the 800,000 documents of the RCV1-v2 data collection. Preparation: The data were formatted in a bag-of-words representation. The representation uses 47,236 unique stemmed tokens, see http://jmlr. org/proceedings/papers/ v27/supplemental/datasetsutl12a. pdf for details. We considered all levels of the hierarchy to select the most promising categories. Representation: Bag-of-word features. SET 4. 5: YOLANDA This dataset was prepared by T. Bertin-Mahieux from original data provided by This data is a subset of the Million Song Dataset: http://labrosa. ee. columbia. edu/millionsong/ a collaboration between LabROSA (Columbia University) and The Echo Nest. Prepared by T. Bertin-Mahieux. This version was obtained from C. J. Lin. : "YearPredictionMSD Data Set " (https://archive.
ics. uci. edu/ml/datasets/YearPredictionMSD). Past Usage: The Million Song Dataset. Thierry Bertin-Mahieux, Daniel P. W. Ellis and Brian Whitman, Paul Lamere. https://archive. ics. uci. edu/ml/datasets/ YearPredictionMSD; http://ismir2011. ismir. net/papers/OS6-1. pdf. Description: Prediction of the release year of a song from audio features. Songs are mostly western, commercial tracks ranging from 1922 to 2011, with a peak in the year 2000s. Preparation: The data were obtained by C J Lin. They respect the original representation. Representation: Features: 12 = timbre average, 78 = timbre covariance. ROUND 5 SET 5. 1: ARTURO This dataset was prepared by Sergio Escalera from original data provided by Sergio Escalera, Xavier Baro, Jordi Gonzalez, Miguel A. Bautista, Meysam Madadi, Miguel Reyes, V?ctor Ponce: "Multimodal Gesture Recognition Data set using audio features" (http: //sunai.
uoc. edu/chalearn/). Past Usage: The data from which the ABGR was generated have been used by several people in two challenges (Multimodal Gesture Recognition and Looking at People Challenges), the number of registered participants exceeded 200 hundred (al least 20 people participated throughout the final stages and developed highly competitive methods). More information can be found in: Sergio Escalera et al. Multi-modal Gesture Recognition Challenge 2013 : Dataset and Results. Proc. of ICMI 2013 , pp. 445-452, 2013, and   Preparation: For the purpose of the AutoML challenge, all samples were merged. Frames of the clip were first described by the 13 MEL coefficients extracted from the audio signal, and then clustered o generate a 200-words vocabulary, which was used to represent the videos.
Representation: Bag-of-Visual-Words using Mel coefficients coordinates, vocabulary of 200 codewords was considered. SET 5. 2: CARLO Preparation: For the purpose of the AutoML challenge, all samples were merged and the data were freshly randomly split in three sets: training, validation, and test. The order of the features (pixels) was also randomize, after adding a few distractor features (probes) that are permuted versions of real features. Representation: neuronal connection. SET 5. 3: MARCO This dataset was prepared by Yin Aphinyanaphongs from original data provided by William Hersh: "Ohsumed TEXT dataset" (http://ir. ohsu. edu/ohsumed/ohsumed. html). Past Usage: Many studies have used the ohsumed corpora for information retrieval research in the biomedical literature. See http://scholar.
google. com/scholar?es_sm= 91&um=1&ie=UTF-8&lr=&cites=13802943827211985373 for a listing of papers that cite this work. Description: See the dataset url for more information. To summarize, these are biomedical articles from 1987 to 1991 from over 270 medical journals from the primary literature that contain titles, abstracts, human-assigned MeSH terms, publication types, authors, and source. Preparation: The original dataset contains 348,566 references from MEDLINE. , the on-line medical information database, consisting of titles and/or abstracts from 270 medical journals over a five-year period  (1987) (1988) (1989) (1990) (1991). The available fields are title, abstract, MeSH indexing terms, author, source, and publication type. We applied the following steps in B", B causes A", "A and B are independent" or "A and B are dependent but not causally related".
Preparation: One of the participant extracted features of the joint distribution of the variable pairs. Those feature (which we provide), include information theoretic features such as conditional entropy and results of independence tests. Representation: Features. Appendix C. Datasets of the 2018 AutoML challenge Description: The task of ADA is to discover high revenue people from census data. This is a two-class classification problem. The raw data from the census bureau is known as the Adult database in the UCI machine-learning repository. The 14 original attributes (features) include age, workclass, education, education, marital status, occupation, native country, etc. Categorical features were eliminated and the original numerical features were preprocessed to obtain 48 attributes. Preparation: A set of reasonably clean records was extracted using the following conditions: ((AAGE > 16) and (AGI > 100) and (AF N LW GT > 1) and (HRSW K > 0)).
Representation: The features include age, workclass, education, etc. SET 1. 2: ARCENE This dataset was prepared by Isabelle Guyon. The associated task is handwritten digit recognition. Specifically, the problem of separating the odd numbers from even numbers. This is a twoclass classification problem with sparse continuous input variables, in which each class is composed of several clusters. It is a problems with heterogeneous classes. Past Usage: It was used previously used in the Performance Prediction challenge, the Model Selection game, and the Agnostic Learning vs. Prior Knowledge (ALvsPK) challenge. Description: The dataset was formed with instances from the MNIST dataset that is made available by Yann LeCun at http://yann. lecun. com/exdb/mnist/. Preparation: The following process was followed for preparing the data: Pixels that were 99% of the time white were removed.
This reduced the original feature set of 784 pixels to 485. The original resolution (256 gray levels) was kept. The feature names are the (i,j) matrix coordinates of the pixels (in a 28x28 matrix). Two digit numbers were generated by dividing the datasets into to parts and pairing the digits at random. The task is to separate odd from even numbers. The digit of the tens being not informative, the features of that digit act as distracters. Representation: Pixels from the images were used as features. More information on the dataset can be found in the following link: http://clopinet. com/isabelle/ Projects/agnostic/Dataset. pdf SET 1. 4: GUILLERMO This data set was prepared by Luis Pellegrin and Hugo Jair Escalante. It comprises preprocessed image-text pairs. Original data was obtained from the SAIAPR TC12 benchmark, provided and prepared by Michael Grubinger and Hugo Jair Escalante (http:// imageclef.
Representation: Images were represented by the response of a pretrained CNN (penultimate layer of VGG-16). Words were represented by their Word2Vec representation. An embedding of 200 dimensions was considered, the embedding was trained with the Wikipedia collection. SET 1. 5: RL This is a confidential dataset provided by the 4paradigm company, hence we cannot disclose confidential information about it. Although this dataset is publicly available as it was used for the feedback phase of the 2018 AutoML challenge. Past Usage: This data set was specifically generated for the 2018 AutoML challenge. Description: The RL data set is associated to a real-world recommendation task involving real users. Items can be: video, audio and activities recommendations, and labels are generated by clicks from users.
Instances in this dataset are chronologically ordered, real recommendations and clicks of users from a small time period were considered. Preparation: A small sample from real recommendations-clicks was taken for preparing this data set. The class imbalance ratio for this dataset was determined to resemble the actual imbalance ratio observed in practice in the associated recommendation task. Representation: Processed numerical and categorical features encoding descriptive information were made available with this data set. This is a confidential dataset provided by the 4paradigm company, hence we cannot disclose confidential information about it. Past Usage: This data set was specifically generated for the 2018 AutoML challenge. Description: The PM data set is associated to a real-world click prediction task involving real users.
More specifically a search-result-click through rate-prediction problem is considered. Instances in this dataset are chronologically ordered, real clicks of users from a small time period were considered. Preparation: A small sample from real search-results-clicks was taken for preparing this data set. The class imbalance ratio for this dataset was determined to resemble the actual imbalance ratio observed in practice in the associated recommendation task. This is a confidential dataset provided by the 4paradigm company, hence we cannot disclose confidential information about it. Past Usage: This data set was specifically generated for the 2018 AutoML challenge. Description: The RH data set is associated to a real-world recommendation task involving real users. Items can be: video, audio and activities recommendations, and labels are generated by clicks from users.
Instances in this dataset are chronologically ordered, real recommendations and clicks of users from a small time period were considered. Preparation: A small sample from real recommendations-clicks was taken for preparing this data set. The class imbalance ratio for this dataset was determined to resemble the actual imbalance ratio observed in practice in the associated recommendation task. Representation: Processed numerical and categorical features encoding descriptive information were made available with this data set. This is a confidential dataset provided by the 4paradigm company, hence we cannot disclose confidential information about it.
SET 2. 3: RI Past Usage: This data set was specifically generated for the 2018 AutoML challenge. Description: The RI data set is associated to a real-world recommendation task involving real users. Items can be: video, audio and activities recommendations, and labels are generated by clicks from users. Instances in this dataset are chronologically ordered, real recommendations and clicks of users from a small time period were considered. Preparation: A small sample from real recommendations-clicks was taken for preparing this data set. The class imbalance ratio for this dataset was determined to resemble the actual imbalance ratio observed in practice in the associated recommendation task. Representation: Processed numerical and categorical features encoding descriptive information were made available with this data set.
This data set was prepared by Luis Pellegrin and Hugo Jair Escalante. It comprises preprocessed image-text pairs. Original data was obtained from the common objects in context collection (http://cocodataset. org/). steps, including feature standardization, sample normalization, and replacement of missing values. This is consistent with the frequent use of ensembles of decision trees based on decision thresholds, which do not require complex preprocessing. Other preprocessing steps included grouping modalities for categorical variables (20%) and discretization (4%). Few participants also reported having used non-linear transforms such as log. Most participants did not perform any feature engineering, which can largely be explained by the fact that they did not know the application domain of the data sets.
We are at least assured that they delivered results in less than 20 minutes on every data set, because this was the time limit for the execution. Most respondents claimed to have developed a simple method, easy to implement and parallelize (75% used multi-processor machines, 32% used algorithms run in parallel on different machines), but few claimed that their method was original or principled, and most relied on third-party libraries; scikit-learn, which was used in the starting kit, was frequently used. Luckily, this also resulted in code that was made available as open source-with only 10% exceptions. Python was used by 82% of the respondents. This is also explained by the fact that the starting kit was in Python. Although Codalab allows users to submit any Linux executable, the organizers provided no support for this.
Even then, 25% used at least one of the following languages: C/C++, Java, or R, sometimes in combination with Python. The fact that the Codalab backend ran on Linux may also explain that 86% of the respondents ran on Linux; others used Windows or MacOS. Memory consumption was generally high (more than half of the respondents used between 8 and 32 GB, and 18% used more that 32 GB). Indeed, when we introduced sparse data in Round 3, the sample code was memory demanding and we had to increase the memory on the server up to 56 GB. Unfortunately, this remained a problem until the end of the challenge-which we traced to an inefficient implementation of the data reader and of Random Forest for sparse matrices.
aad freiburg The open-source solution of AAD Freiburg uses a heterogeneous ensemble of learning machines (auto-sklearn  (Feurer et al. , 2015a,c) ) combining the machine learning library scikit-learn  (Pedregosa et al. , 2011)  with the state-of-the-art SMBO method SMAC to find suitable machine learning pipelines for a data set at hand. This is essentially a reimplementation of Auto-WEKA. To speed up the optimization process they employed a meta-learning technique  (Feurer et al. , 2015b)  which starts SMAC from promising configurations of scikit-learn. Furthermore, they used the outputs of all models and combined these into an ensemble using ensemble selection. Their latest version uses a python reimplementation of SMAC  (Hutter et al. )  of Bayesian Optimization with Random Forests applied to a flexible configuration space describing scikit-learn.