AutoAI implementation details
AutoAI implementation details AutoAI automatically prepares data, applies algorithms, or estimators, and builds model pipelines that are best suited for your data and use case. The following sections describe some of these technical details that go into generating the pipelines and provide a list of research papers that describe how AutoAI was designed and implemented. * [Preparing the data for training (pre-processing)](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=endata-prep) * [Automated model selection](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=enauto-select) * [Algorithms used for classification models](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=enestimators-classification) * [Algorithms used for regression models](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=enestimators-regression) * [Metrics by model type](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=enmetric-by-model) * [Data transformations](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=endata-transformations) * [Automated Feature Engineering](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=enfeat-eng) * [Hyperparameter optimization](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=enhyper-opt) * [AutoAI FAQ](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=enautoai-faq) * [Learn more](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=enadd-resource) Preparing the data for training (data pre-processing) During automatic data preparation, or pre-processing, AutoAI analyzes the training data and prepares it for model selection and pipeline generation. Most data sets contain missing values but machine learning algorithms typically expect no missing values. On exception to this rule is described in [xgboost section 3.4](https://arxiv.org/abs/1603.02754). AutoAI algorithms perform various missing value imputations in your data set by using various techniques, making your data ready for machine learning. In addition, AutoAI detects and categorizes features based on their data types, such as categorical or numerical. It explores encoding and scaling strategies that are based on the feature categorization. Data preparation involves these steps: * [Feature column classification](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=encol-classification) * [Feature engineering](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=enfeature-eng) * [Pre-processing (data imputation and encoding)](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-details.html?context=cdpaas&locale=enpre-process) Feature column classification * Detects the types of feature columns and classifies them as categorical or numerical class * Detects various types of missing values (default, user-provided, outliers) Feature engineering * Handles rows for which target values are missing (drop (default) or target imputation) * Drops unique value columns (except datetime and timestamps) * Drops constant value columns Pre-processing (data imputation and encoding) * Applies Sklearn imputation/encoding/scaling strategies (separately on each feature class). For example, the current default method for missing value imputation strategies, which are used in the product are most frequent for categorical variables and mean for numerical variables. * Handles labels of test set that were not seen in training set * HPO feature: Optimizes imputation/encoding/scaling strategies given a data set and algorithm Automatic model selection The second stage in an AutoAI experiment training is automated model selection. The automated model selection algorithm uses the Data Allocation by using Upper Bounds strategy. This approach sequentially allocates small subsets of training data among a large set of algorithms. The goal is to select an algorithm that gives near-optimal accuracy when trained on all data, while also minimizing the cost of misallocated samples. The system currently supports all Scikit-learn algorithms, and the popular XGBoost and LightGBM algorithms. Training and evaluation of models on large data sets is costly. The approach of starting small subsets and allocating incrementally larger ones to models that work well on the data set saves time, without sacrificing performance. Snap machine learning algorithms were added to the system to boost the performance even more. Selecting algorithms for a model Algorithms are selected to match the data and the nature of the model, but they can also balance accuracy and duration of runtime, if the model is configured for that option. For example, Snap ML algorithms are typically faster for training than Scikit-learn algorithms. They are often the preferred algorithms AutoAI selects automatically for cases where training is optimized for a shorter run time and accuracy. You can manually select them if training speed is a priority. For details, see [Snap ML documentation](https://snapml.readthedocs.io/). For a discussion of when SnapML algorithms are useful, see this [blog post on using SnapML algorithms](https://lukasz-cmielowski.medium.com/watson-studio-autoai-python-api-and-covid-19-data-78169beacf36). Algorithms used for classification models These algorithms are the default algorithms that are used for model selection for classification problems. Table 1: Default algorithms for classification Algorithm Description Decision Tree Classifier Maps observations about an item (represented in branches) to conclusions about the item's target value (represented in leaves). Supports both binary and multiclass labels, and both continuous and categorical features. Extra Trees Classifier An averaging algorithm based on randomized decision trees. Gradient Boosted Tree Classifier Produces a classification prediction model in the form of an ensemble of decision trees. It supports binary labels and both continuous and categorical features. LGBM Classifier Gradient boosting framework that uses leaf-wise (horizontal) tree-based learning algorithm. Logistic Regression Analyzes a data set where one or more independent variables that determine one of two outcomes. Only binary logistic regression is supported Random Forest Classifier Constructs multiple decision trees to produce the label that is a mode of each decision tree. It supports both binary and multiclass labels, and both continuous and categorical features. SnapDecisionTreeClassifier This algorithm provides a decision tree classifier by using the IBM Snap ML library. SnapLogisticRegression This algorithm provides regularized logistic regression by using the IBM Snap ML solver. SnapRandomForestClassifier This algorithm provides a random forest classifier by using the IBM Snap ML library. SnapSVMClassifier This