Comprehensive Guide to Machine Learning (Part 3 of 3)
By: Tapas Das
Tags: Machine Learning, Optuna, K Fold, Catboost, Hyperparameter Tuning

7) Baseline model building

The first step towards model building is to decide that which machine learning framework to use, based on the type of task we have at our hand. Below are few of the machine learning frameworks which work well in majority of the scenarios:

Neural Networks (Tensorflow/ Keras/ PyTorch)

XGBoost

LightGBM

CatBoost

Logistic Regression

Random Forest

Support Vector Machines (SVM)

The “Pet Adoption” dataset that I have been referring to through out this series, has majority of features as categorical variables. So I decided to build the final machine learning model using “CatBoost” framework, since it works pretty well with categorical features.

For experiment sake, I also tried out models based on Neural Networks, XGBoost and LightGBM. But CatBoost gave me the best results out of all these machine learning frameworks. You can find the other codebase at below link.

Below image shows the final CatBoost model I used for making predictions.

Let’s go over few of the crucial model configurations for better understanding of different nuts and bolts of CatBoost framework.

objective='MultiClass'

In the “Pet Adoption” dataset, we are classifying between 4 different breed categories and 3 different pet categories. So considering all that, we need to set the model objective as “MultiClass”.

eval_metric='TotalF1'

We need to set an evaluation metric for the CatBoost model, so that we can judge that the model is behaving as expected and converging to global minima. For classification tasks, below are the commonly used evaluation metrics:

Accuracy

F1 Score

Area Under the Curve (AUC)

Precision Score

Recall Score

Generally for highly imbalanced datasets like “Pet Adoption”, it’s preferred to use F1 Score or AUC as evaluation criteria, since they provide a much better holistic understanding of the model performance. You can get further understanding of these metrics by visiting below link.

class_weights=[0.165, 0.185, 1]

In case of imbalanced datasets, the machine learning models tend to get biased towards the majority class. To ensure that the model is giving proper weightage to all the prediction classes, we can set the class weights.

Below example shows how to calculate the class weights for imbalanced datasets.

Total Sample: 100

Class A: 60 samples

Class B: 30 samples

Class C: 10 samples

Class A %: (60/100) = 0.6

Class B %: (30/100) = 0.3

Class C%: (10/100) = 0.1

Class Weight = (Lowest Class %) / (Current Class %)

Class A weight: 0.1/0.6 =0.1667

Class B weight: 0.3/0.2 = 0.5

Class C weight: 0.1/0.1 = 1

learning_rate=0.025

This hyper-parameter sets the speed at which the model will converge towards the global minima.

Learning rate is the most critical hyper-parameter in any machine learning model. Setting it too low will cause the model to train very slowly. On the other hand, setting it to high value can cause the model to overshoot the global minima and result in poor predictions.

I’d highly recommend to go through below link to get deeper understanding of learning rate and it’s significance in machine learning models.

reg_lambda=0.009

This parameter helps in preventing model overfitting, which we have already discussed in the 2nd post of this series.

You can visit below for better understanding of rest of the CatBoost model parameters.

Also, I’d highly recommend to go through below link for better understanding of ensemble machine learning models (CatBoost, XGBoost, LightGBM).