A Review of Azure Automated Machine Learning (AutoML)
By: Luca Zavarella
Tags: Python, Machine Learning, Azure Machine Learning, Automl, Azure

Azure AutoML is a cloud-based service that can be used to automate building machine learning pipelines for classification, regression and forecasting tasks. Its goal is not only to tune hyper-parameters of a given model, but also to identify which model to use and how to pre-process the input dataset.

fig. 1 — Azure AutoML workflow taken from the official infographic

AutoML uses Bayesian optimization to identify better hyper-parameters than human experts. It also uses collaborative filtering (Probabilistic Matrix Factorization) to search for the most promising data transformation pipelines efficiently, based on a database that is built by performing millions of different pipeline evaluation experiments on many datasets. This database helps in finding the good solutions for new datasets quickly.

Azure Machine Learning offers two experiences for working with AutoML:

A simple user interface included into Azure ML Studio for limited/no code experience customers

The Azure ML Python SDK, a Python library that allows code experience customers and Data Scientists to develop ML models using all the features Azure ML makes available

All features described in the following paragraphs are those provided by the platform on December 20, 2020.

Using AutoML via User Interface

The first step required by the AutoML UI is to choose a previously registered Dataset:

fig. 2 — Azure ML Studio, Automated ML section

Then, information about the Experiment to use, the target variable to consider and the Compute Cluster to use for training are required:

fig. 3 — Azure ML Studio, Configure the new Automated ML

Next, the information about what kind of ML problem has to be solved, its configurations and featurization settings are required.

Additional Configuration Settings

It’s important to select the AutoML configuration settings that fit the problem to be solved.

fig. 4 — Azure ML Studio, Configure the new Automated ML task type

The available configurations are the following:

fig. 5 — Azure ML Studio, Configure Automated ML additional settings

A primary metric has to be specified in order to optimize the model using hyperparameter tuning. The available primary metrics for each kind of supported ML problem are the following:

fig. 6 — AutoML primary metrics available

As a Python user will probably have already noticed, these metrics are a subset of the ones provided in the scikit-learn metrics.

Specific algorithms could be blocked during the training phase using a black-list. The supported algorithms will be detailed in the next paragraph.

Azure AutoML is an automated process (by definition). It could be time consuming and not cheap to find the best model for not trivial datasets. That’s why an early stopping policy can be set. Two options are available:

The maximum amount of training time (in hours).

The primary metric score threshold, reached which the AutoML job stops.

The validation type can also be chosen. Here the available options.

Auto

K-fold cross-validation

Monte Carlo cross-validation (also called Shuffle Split or Random Permutation Cross-Validator)

Train-validation split (the classic hold-out validation)

If auto is chosen, AutoML will perform a different type of validation according to the number of rows of the dataset. In details:

For datasets larger than 20,000 rows, the 10% of the initial training data is taken as the validation set. In turn, that validation set is used for metrics calculation.

For datasets smaller than 20,000 rows, the cross-validation approach is applied: 10 folds will be used if the dataset is less than 1000 rows; 3 folds otherwise.

For time-series forecasting is available only one type of validation: the Rolling Origin Cross-Validation (ROCV). It divides the series into training and validation data using a time origin point. By sliding the origin through time, cross-validation folds are generated.

fig. 7 — Visualization of the Rolling Origin Cross-Validation (ROCV)

This strategy preserves the integrity of the time series data and eliminates the risk of data loss.

Finally, the option to define the maximum number of iterations that should be executed in parallel during the AutoML training phase has to be set (Max concurrent iterations). This should be less than the number of nodes of the compute target (training cluster), with a maximum value of 100. For optimal parallelism, the number of parallel iterations must be equal to the number of nodes in the training cluster. Evidently, the more nodes there are in the cluster, the more you can parallelize the execution.

Featurization Settings

Featurization is the act of applying techniques of Feature Engineering to the dataset, that is to create features that help machine learning algorithms to learn better.

fig. 8 — Azure ML Studio, Enable deep learning

The “Enabling deep learning” option can be misleading. Deep learning algorithms are used right now only for text-based featurizations in classification problems. Depending on the scenario, BERT deep learning architecture for text data featurization in 100 languages is used on GPU nodes, or Bidirectional LSTM (BiLSTM) on CPU nodes. Moreover, if “enable deep learning” is set, validation is limited to train_validation split.

Actually, the only training algorithm using DNN is ForecastTCN, an algorithm developed by Microsoft designed to tackle the most demanding forecasting tasks, capturing nonlinear local and global trends in the training dataset.

Clicking on “View featurization settings”, featurization options can also be set. Automatic featurization can be enabled or disabled and some settings can be customized. First of all, a subset of the dataset’s features can be selected to be used in the training phase:

fig. 9 — Azure ML Studio, Featurization settings 1

Then, the type and the imputing action can be chosen for each feature:

fig. 10 — Azure ML Studio, Featurization settings 2

Feature types are quite obvious except for a string feature. In this case it has to be specified to be one of the following type:

Categorical: the entire text string appearing in the feature is defined as a category and behind the scene the feature is transformed in an indicator vector; often used for low cardinality categorical features

Categorical hash: its values are considered categorical and converted into an indicator array by hashing the value and using the hash as an index in the bag; often used with categorical features having high cardinality

Text: raw text to be transformed later appropriately

Specific data imputation techniques can be selected for each feature. Unfortunately, to date only basic techniques are implemented: fill with mean, median, most frequent value and fill with constant value.

At this point, clicking on the “Finish” button, the AutoML training run will be launched.

Using AutoML via Python SDK

Automated machine learning runs can be launched on a local machine or on a remote one (Azure ML compute cluster or Azure Databricks cluster). The data to be used as training dataset must be on the local machine (only for local AutoML runs) or in the cloud (Azure Blob Storage, Azure File Share, Azure Data Lake Storage, Azure SQL Database, Azure PostgreSQL Database and Azure MySQL Database). Moreover, the data can be read into a Pandas DataFrame (just for local AutoML) or an Azure Machine Learning TabularDataset (local and remote AutoML).

The main Python class used to launch an AutoML run is the AutoMLConfig one. This configuration object allows data scientists to set the configuration of the experiment execution, as well as the training data to be used at run time.

An example of working Python code is the following:

from azureml.train.automl import AutoMLConfig automl_settings = { "n_cross_validations": 10,

"primary_metric": 'AUC_weighted',

"enable_early_stopping": True,

"max_concurrent_iterations": 10,

"max_cores_per_iteration": -1,

"experiment_timeout_hours": 1,

"featurization": 'auto',

"enable_dnn": True,

"verbosity": logging.INFO } label_column_name = 'fraud_reported' automl_config = AutoMLConfig( task = 'classification',

debug_log = 'automl_errors.log',

compute_target = compute_target,

training_data = training_data,

label_column_name = label_column_name,

**automl_settings ) remote_run = experiment.submit(automl_config, show_output = False) remote_run

Details about the various uses of AutoMLConfig will not be discussed here. Among the other parameters, it is worth saying that it is also possible to customize the use of featurizations thanks to the FeaturizationConfig class, in order to define specific feature engineering strategies. The following setting can be specified:

Add or remove column purpose in order to override the feature type for specified columns

Add or remove transformer parameters in order to customize transformers like Imputer, HashOneHotEncoder and TfIdf (see the next paragraph for details)

Block specific transformers

Add a drop column to ignore for featurization and training

Here is a non-exhaustive example:

from azureml.automl.core.featurization import FeaturizationConfig featurization_config = FeaturizationConfig() # Drop columns before training

featurization_config.drop_columns = ['policy_number', 'incident_location_num'] # Define categorical features

featurization_config.add_column_purpose('policy_state', 'Categorical') featurization_config.add_column_purpose('insured_zip', 'CategoricalHash') featurization_config.add_column_purpose('insured_occupation', 'CategoricalHash') # Define imputers

featurization_config.add_transformer_params('Imputer', ['Quantity'], {"strategy": "constant", "fill_value": 0}) # Block specific transformers

featurization_config.blocked_transformers = ['LabelEncoder']

Featurization Details

One of the most interesting part in Azure AutoML is the featurization. It consists in the set of data-scaling and normalization techniques, plus some feature engineering processes, automatically applied to the input dataset.

The scaling and normalization processes to help algorithms perform well are the following:

StandardScaleWrapper

MinMaxScalar

MaxAbsScaler

RobustScalar

PCA

TruncatedSVDWrapper

SparseNormalizer

More details can be found here.

The feature engineering processes can be summarized in the following points:

fig. 11 — AutoML feature engineering strategies

More details can be found here and here.

Supported Algorithms

There are more than one algorithm supported by Azure AutoML depending on the type of problem to be addressed.

Below is an exhaustive list of supported algorithms at the time of this writing. If a model has to be run on a variety of platforms and devices, converting it in the ONNX format could be the right solution. In that case, only those algorithms indicated with an * are able to be converted to the ONNX format.

fig. 12 — AutoML supported algorithms

Data Guardrails

AutoML is a fully automated process that cannot best solve all data problems that your input dataset may contain. For this reason Data Guardrails can help the data scientist to identify potential issues with data and to take corrective actions for improved results.

Currently supported data guardrails can be found here. One of the most useful is the Class balancing detection. Here an example taken from the UI:

fig. 13 — AutoML data guardrails

with its details:

fig. 14 — AutoML data guardrails additional details

Here instead an example taken using the Python SDK:

fig. 15 — AutoML data guardrails through Python SDK

Ensembling Models

Ensemble modeling is a process where multiple diverse models are created with the aim to aggregate the predictions of each base model, resulting in one final prediction for the unseen data that is usually more accurate than the each one returned by each single model. Basically, ensembling strategies reduce the generalization error of the prediction.

Currently Azure AutoML supports two types of ensemble models: VotingEnsemble and StackEnsemble (taken from scikit-learn ensamble methods). In particular, the following implementations are used:

Voting : soft-voting which uses weighted averages

: soft-voting which uses weighted averages Stacking: two layers are used, where the first layer has the same models as the voting ensemble, and the second layer model is used to find the optimal combination of the models from the first layer.

fig. 16 — Example of ensemble models returned by AutoML

They are enabled by default, but can easily disabled using the AutoMLConfig class.