Comparing AutoML/Non-Auto-ML Multi-Classification Models
By: Zeineb Ghrib
Tags: Scikit Learn, Automl, Machine Learning, Data Science, Previsionio

Auto-ML approach:

Let’s get the dataset:

import pandas as pd

df = pd.read_csv('winequality-white.csv', sep=';')

Create train / test subsets

Lets create a sub-sample (about 20% of the overall dataset) that we will use as a holdout data-set, in order to evaluate the generalization error of our models. this sub-sample will be put aside, and not used for training. Hence, we will find out how well our models will perform on new data (not seen during the training phase).

It exist many ways to create the sample, the simplest is to use train_test_split() of scikit learn

from sklearn.model_selection import train_test_split

train_set, test_set = train_test_split(df, test_size=0.2, random_state=42)

During Feature engineering step you can construct two types of features:

Business derived features : example here if we have some knowledge in chemistry in chemistry we can extract new feature from a combination of fixed acidity, volatile acidity and citric acid to create a new explicative feature

: example here if we have some knowledge in chemistry in chemistry we can extract new feature from a combination of fixed acidity, volatile acidity and citric acid to create a new explicative feature Transformation based features : these features are derived from ML operations such as scaling, encoding, normalization, ACP components… to create new features more valuable to the models.

The second kind of feature engineering is supported by the platform: once you launch the use case on your dataset, you can select transformations that you want to apply on your dataset, and they will be automatically computed and added as stand-alone features to get more information about the feature-transformation supported by the platform consult this link

Launch auto-ml pipeline within 10 code lines:

Step 0 : Connect to your instance via the sdk

import previsionio as pio

import pandas as pd



URL = 'https://XXXX.prevision.io'

TOKEN = '''YOUR_MASTER_TOKEN'''



# initialize client workspace

pio.prevision_client.client.init_client(URL, TOKEN)

Check my previous post to know how to get your MASTER_TOKEN

Step1 : Register the train/test subsets as Prevision Data sets

To launch a new use case you need to stock them into your work-space:

pio_train = pio.Dataset.new(name=’white-wine-train’, dataframe=train_set)

pio_test = pio.Dataset.new(name=’white-wine-test’, dataframe=test_set)

Step2 : Dataset and training configuration

To launch a new use case, you need to define some configuration parameters, and then simply use the SDK’s BaseUsecase derived methods to have the platform automatically take care of starting everything for you. Example here we need tell to the platform that the target column is quality

col_config = pio.ColumnConfig(target_column=’quality’)

Now we will also specify some training parameters, such as which models are used, which transformations are applied, and how the models are optimized..



uc_config = pio.TrainingConfig(models=[pio.Model.LinReg, pio.Model.LightGBM, pio.Model.RandomForest],

simple_models=[],

features=pio.Feature.Full.drop(pio.Feature.PCA,

pio.Feature.KMeans,

pio.Feature.PolynomialFeatures),



profile=pio.Profile.Quick)

To get the exhaustive list of `previsionio.ColumnConfig` and `previsionio.TrainingConfig` parameters checkout this Churn tutorial.

Step3: Launch a multi-classification use case

Now we will use the `fit()` method of the `previsionio.MultiClassification` class: we select the log loss as a performance metric : Here a good post to understand the log_loss and binary cross-entropy metrics

# now we will launch

# launch Multiclassif auto ml use case

uc = pio.MultiClassification.fit('wine_quality_from_sdk',

dataset=pio_train,

metric=pio.metrics.MultiClassification.log_loss,

holdout_dataset=pio_test,

column_config=col_config,

training_config=uc_config,

with_blend=False)

You can visit this post to get more information about the `fit()` options

Result analysis

For a multi classification use case, Prevision supports the OvA strategy (one-versus-all). The platform would generate probability for each modality of the target column and the selected one is the argmax.

let’s find-out the best performances found by the platform:

print("{} models were trained on this usecase:

".format(len(uc)))

for model in uc.models:

print(model.name,' === ' ,'%.3f'%model.score, 'log_loss')



print("

##################################################")

print('The best model found in this use case is a', best.algorithm)

print("the best cross validation performance is {} of log_loss".format('%.3f'%best.score)) print("

##################################################")

print('The best model performance in the holdout-dataset is ', '%.3f'%uc._status['holdoutScore'])

=> The best model is a random forest and it’s cross val score is 0.93

Now we will train a random forest with scikit learn API