Comprehensive Guide for Machine Learning Competitions
Tags: Kaggle, Machine Learning, Guides And Tutorials, Sklearn, Competition

Getting Started !

One of the most important thing in Machine Learning is to understand the data. Most of the online competitions are based on supervised Machine Learning where some kind of prediction needs to be done.

Now this prediction could be a :

1) Regression problem where the output label is a continuous range of integers.

Examples include “predicting the age of the person”, “predicting the price of an item”, “census income”, “compressive strength of materials” etc.

2) Classification problem where the output label is discrete and finite in nature.

Examples include “predicting the gender of the person”, “predicting the success of a startup”, “object recognition”, “text classification”, “sentiment analysis”, “cat vs dog classification” etc.

Still getting confused, let’s take a simple example.

If you are asked to predict the prices of stocks, take a look at the output column, if there are a wide range of values like 6.2, 7.1, 8.8, 9,10 … and predicting “6.5" (something not mentioned in the output column of the training set) is also a possibility, you could safely say that this is a classic regression based problem. If you are asked to predict the direction of stock and the output label contains +1, 0 and -1 as the sole outputs where “+1” denotes upward rise, “ -1” denotes downward fall and “0" denotes no change. Here, we can clearly notice there are only 3 possibilities for the output label and predicting “6.5” as the output makes no sense. Hence, this becomes a classification based problem wherein each output label infers a special meaning.

Once you have decided the type of algorithms you could use for the given problem, things become systematic and easy.

Loading the Data

In this article we would be considering structured data (highly organized data in excel sheets, csv files etc.). For dealing with Unstructured data (audio, text, images, video files), Deep Learning models take an edge. I will be covering in the upcoming articles.

Pandas is the most commonly used library to load “csv files” and “excel sheets”, as pandas provides lots of powerful data manipulations and aggregations.

Link to the dataset. [view]

With a single line of code, you could read the data present in the file “training_data.csv”

With the help of “data.head()” you could have a look at the rows and columns of the data to estimate the type of Machine Learning algorithm to be applied as discussed earlier [regression or classification].

Exploratory Data Analysis

Once the data is loaded, the next step is to explore the data and check for irregularities if present.

Most of the times, we could observe some missing values in each column of our dataset.

By executing the above line, if we see 0 corresponding to all the columns that means there are no missing values.

If you observe some null values in any column, we need to take care of those particular inputs as the model won’t be able to make sense out of null values.

It’s just numerical values that the Machine understands !

There is a huge difference between “0” and “None”, “0” is the value of that particular feature while “None” implies absence of a value.

It’s possible that during the data preparation, all the values in the csv files are not filled, some might be left blank. The cells which are left blank takes up “NaN”(Not a Number).

Whenever we encounter NaN / missing values, here are a few ways to deal with it.

If the number of such rows are very less compared to the total rows in the dataset, you could safely drop such rows.

For example: if there are 25 rows have missing values out of 5000 rows in the data, then dropping these rows won’t make much of a difference.

Impute the “NaN” values with some specific values.

We could make use of the SimpleImputer class present in the sklearn library, one of the most famous library for Machine Learning implementations.

Since, we are dealing with missing values ie. “NaN”, so in the first parameter we will be mentioning np.nan.

For strategy, we have 4 options : “mean”, “median”, “most_frequent”, and “constant”.

Note: “mean” and “median” are statistical measures and therefore could be used only with numerical data. “most_frequent” and “constant” could be used with both numerical as well as categorical data.

After this step, we would be having clean data with no null values.

Tackling Numerical Features

We could observe that the max value of the feature “Age” is 60 whereas the max value of “DailyRate” is 1499. We can clearly notice a huge difference in the range of values.

Machine Learning involves elementary addition and multiplication operations, basically there is a weightage corresponding to each feature. Due to obvious reasons, a feature smaller range of values will be assigned more weightage to take its affect into account. This would lead to a model which is not interpretable in terms of feature importance and ranking. Do not scale the Output Feature, doing so will lead to disastrous results. Only the input features need to be scaled.

A simple solution would be to scale the values so that all the numerical values span a similar range of values.

Using the MinMaxScaler class from sklearn library we have scaled all the numerical columns to range from [0, 1].

Tackling Categorical Features

We could observe the unique values corresponding to each categorical feature. Based on the number of values and the ordinal nature of the values certain methods could be put into action.

Machine Learning models cannot interpret string values, so we need to encode these categorical features into numerical values.

You can make use of pandas map function to manually encode the features.

The simplest way to deal with this problem is to use LabelEncoder. It will start assigning integer values starting from 0 to each new feature label.

Alternatively, you can also one hot encode the values, when you feel all the particular values should receive equal weightage and no special preference is given to a particular category of values.

If you are getting confused, I would suggest to move forward with LabelEncoder. In case, the output feature is also in the form of string values, then you need to encode the output feature / target column as well.

Well done, your data is now ready to be fed into various Machine Learning Models.

Before: missing values, no scaling and encoding

After: Looks systematic !

Getting Data Ready for Model fitting

‘X’ would be the input features that would be fed to the model during training phase. ‘y’ is the corresponding output labels.

Our aim is given a new set of input feature values, our model should predict the most accurate value for the target feature.

Here, we could observe that there are 52 input features that would be fed to the model.

After defining ‘X’ and ‘y’, we need to split our data set into 2 parts ie. training and testing set.

The model is trained on the training set and to test the model’s performance, we will evaluate it on the testing set(unseen by the model till now).

Model Training

The current output label is categorical in nature, so it becomes a classification problem as discussed at the start.

Majority of the classification models are present in the sklearn library.

All you need to do is importing the model, instantiating the class with the required set of parameters.

After this step, you simply need to fit the model on the training set.

Making predictions

Once the model is trained, making predictions is very easy, use model.predict() function and in the parameter, provide the input features. Based on the model weights, the model will predict the corresponding output labels.

There are various metrics to evaluate the models, for this article we would be using accuracy_score.

If you want to know more refer to this link.

As we could see above, even though both models perform very well on the training set, they fail to show the same promise for the testing result.

Overfitting vs Underfitting

A short story: Let’s say you gave 100 MCQ questions as the training set for the model, on which the model is trained. To test whether your model is working fine, you asked the same questions on which the model was trained. Now, you might feel the model is doing great. How could you be assured that the model is answering based on knowledge or simple memorization of all answers. To tackle this, we use the testing set, something the model has not seen till now. The outcome of this evaluation would surely confirm whether the model is able to generalize well or has it purely memorized the 100 questions you gave to it.

In technical terms we call this as overfitting vs underfitting. A good model lies in between these 2 extreme conditions.

“A good Machine Learning Algorithm is the one which is able to generalize its observations and hence delivers excellent performance on new unseen data.”

To put it up practically,

Underfitting occurs when the accuracy on the training set itself is low.

Overfitting occurs when the accuracy on the training set is high but low on the testing set.

There are various ways to tackle these issues.

To tackle underfitting, try to increase the number of iterations of model training, increase the depth of tree based models.

If the above 2 methods don’t fetch satisfactory results, go for advanced feature extraction and feature engineering techniques.

PS: You could use KBest features, RFE(Recursive Feature Elimination etc)

For tackling overfitting, try to reduce the depth of tree based models, use regularization.

Mostly regularization would solve the issues else go for the advanced methods.

Hyperparameter Tuning

“No matter how good the performance is, there is always a scope of improvement”.

All the models have various hyperparameters that can be tweaked to optimize the performance. Hyperparameters are the parameters whose values are fixed manually before model training.

Examples include : “number of estimators”, “learning rate”, “no of splits” etc.

This could be a very time consuming step, as the model training takes place several times.

Grid Search Cross Validation is used to find the optimal set of hyperparameters. In the param_grid which is a python dictionary, you can specify the various values to be tried on.

The outcome of the above code would be a model with the best set of hyperparameters amongst the provided values.

However the drawback of using GridSearchCV is that this method will go through all the intermediate combinations of hyperparameters making it computationally very expensive.

Using Randomized Search CV solves the above issues as you can specify the value of “n_iter”. Since the value is currently 5, this means only 5 sets of randomly selected values will be tested.

This increases the chances to obtain the optimal model at the same time its computational complexity is based on the value of “n_iter”.

If you are running short of time, use lower value of “n_iter” through which the results would be obtained quickly, but the chances of getting the optimal set of hyperparameters would be less.

Bonus Part-1 : Creating a pipeline

You might be wondering that since the steps are pretty much sequential, scaling the data and then fitting the model and hyperparameter tuning, why not club them together.

With the help of sklearn pipeline, all of the steps could be combined into a single step.

I already told you, sklearn is an awesome library with lots of awesome functionalities.

Bonus Part-2: Ensemble Learning

In almost every competition, ensemble techniques fetch the best results.

Ensemble learning in simplistic terms means “together”. There’s a common saying, “a crowd is clever together than a clever individual”.

The main causes of error in learning models are due to noise, bias and variance. Ensemble methods help to minimize these factors. Ensemble methods are designed to improve the stability and the accuracy of Machine Learning algorithms.

So here we basically train lots of weak learners and the final result/prediction is taken up by aggregation of the predictions by individual weak learning or by the process of voting akin to real life elections.

Voting

Here, you can combine three models together to fetch the final prediction based on voting.

I find this method pretty useful as we usually train atleast 3 models to have a comparison based on performance. Combining the 3 or more best models together could result into better results.

The performance might not be better, but is at least as good as the best individual model — the decision tree.

This may be because the other two models had a performance around 10% lower than the decision tree.

Voting Classifier is used when the target variable is categorical.

By default voting is ‘hard’. Use the parameter voting = ‘soft’, if you want the average of the predicted probabilities

A few other models to use

Random forest model could be used which is an ensemble of decision trees. It is the most commonly used algorithm.

Alternatively, you could make use of :

Gradient Boosting classifier

Ada Boost Classifier

Categorical Boosting

Here, the default base estimator is decision tree. There is no need to encode the categorical features. The model could be fit directly.